Progress Updates!

What a crazy couple of weeks it has been!

Two thirds of our team has been extremely busy the last couple of weeks given the meteoric rise and continued success of Pesky Penguins. The good news for all of you Anime fans out there is that the one person who did not get sucked into the maelstrom is our AI and Machine Learning expert, who has continued to make steady progress, one breakthrough at a time!

The model featured on October 05 was never able to converge due to a lack of variety in our training data. This was self-inflincted, as we really cleaned up our training set when the older models failed to figure out hands and fingers.

Our latest model attempts to account for both "too much variety," and "insufficient clean examples" by using two training data sets: a "Primary Target" dataset containing our hypercurated images, and a background set of over 1M images intended to introduce "flavor" into the model. This seems to be working extremely well: we're seeing more variety and detail than ever before in our images, and the model has yet to collapse.

I know it's been said before, but we have really high hopes for this model. And if it doesn't work out, you can trust us to keep iterating until we have our final breakthrough! But I think the images below speak for themselves: we're on to something, and we're almost there!

Thanks for sticking with us, Waifu fans. I know it's been a long road, but the result is that we'll have something nobody else has: 100% unique, high quality girls. Stay tuned!

Spooky season is upon us!

The 128d model with VQ we've been working on for the past week or so collapsed, and there was no salvaging it. The good news is that, while training it, we've further refined our training data, and this has enabled us to go back to our 512d model and have it learn on the curated data. This has resulted in the progress you see below, which includes some of our best samples yet!

Hands are coming along; they remain our arch nemesis, but you can see that some girls at least have normal-sized fingers. Others are being more modest about showing their fingers, but their figures are looking quite good!

On the metadata front, because our training data has been purged of catgirls and other kemono-type ladies, we're going to go back and update our metadata generation and some of the rarity info. We'll be replacing ears and tails with height, and we'll be doing some research on what how each vector in our 512-dimensional space affects the resulting image. Our hope is to get to the point where we can use the metadata to tell the generator "this is how this girl should look," but we're a ways off yet.

We're going to continue training with the refined data to see how much further we get with this model. If it continues to improve, then I would say we're getting close to being able to launch the collection!

Thanks for sticking with us! I know this is slow going, but we only want to release a quality collection. We could have put out our Sept 17 collection if we were a cash grab, but we are not. We are fans of the genre, and we want it to be a tribute.

See you in a few days! 👋

The model is progressing, and so is our architecture!

We spent a lot of time over the weekend and this past Monday refining our machine learning architecture. We did this because we received our first bill from AWS last week, and we had blown through $4k USD in half a month 😱

Machine learning is hard, and machine learning is expensive, and if all you're doing is generating faces, it's a much lighter lift. But we here at Tokimeki Tokens are dedicated to bringing you the finest anime girls, and having believable bodies with outfits and poses is necessary. Not just for the sake of quality, but because it's part of our roadmap. Which we will reveal at a later date.

So our architecture: We decided to switch from AWS to Google Compute, and we got our model saving off-site at regular intervals so that we could transition to preemptible instances, which are about a third to a quarter of the cost of what we were running up at AWS. Not only that, but the GPU instances we're running now are around 4x faster than when we were training the models locally on a multi-GPU training rig.

We moved our 128-dimensional model (the one with Vector Quantization) up to our new instance and let it grind for a couple days. The progress relative to the previous post is incredible, but we're not anywhere close to convergence yet. We'll post another update near the end of the week!

Good morning, Tokimeki fans!

As we mentioned in our last post, we've started training a new model with different parameters to see if we can get it to converge on the finer details. You may remember that hands are our arch nemesis, and it seems to be a common problem — a lot of other anime generators you can find online also have a hard time with hands. Or they skip them altogether and focus only on faces. We are undeterred: our goal is to give you the highest quality product, and if that means taking more time, so be it.

The good news is that means you get to watch a new model evolve from scratch! For this model we've reduced the vector space (basically: the number of parameters the model can tweak while its learning), and implemented vector quantization, which we are hoping will help us solve the “melty” details we were seeing in our earlier models.

The first image after 1 day of training. You can see that it doesn't look anything like an Anime girl, but it has figured out that hair and skin are different colors, and can approximate the outline of a girl. The second image is after 2 days of training, and you can see how much progress it has made. Things like faces and eyeballs have started to appear, the shape is much more well-defined, and clothes are a different color than skin and hair.

This model still has much further to go, but we will keep you updated on our progress!

Over the weekend, our model continued to improve, but it never quite figured out the finer details. In this model, it would appear as though hands are our Achilles heel.

GANs are interesting in that the progress they make is often two steps forward, one step back: in order to learn a new feature, the model often “forgets” the finer details of the feature space while it optimizes a specific feature (clothing may get sloppy while it figures out hair, for example).

We have a couple hypotheses as to why our model is not converging on hands, and we're going to continue to tweak it to see what we can come up with. For now, we're going to reduce the vector space from 512 dimensions to 128 dimensions and see if we can get it to converge with “simpler” parameters.

For those of you who work with AI and Machine Learning every day, this is probably a familiar sight: train the model, see how it behaves, tweak the parameters, run it again. For those on the outside looking in, it may come as a surprise that machine learning is more art than science. But that fact makes it all the more appropriate for use in NFTs 😊

The goal, after all, is to give you one-of-a-kind, entirely unique, high quality anime girls. So we'll keep turning the crank and making progress one day at a time!

It's the first day of training on our improved GPU cluster, and after only one day, we're seeing marked improvements!

Faces and hair are starting to look really good, but clothing and hands are still a little difficult for the GAN. It's made great progress on arms, and the shading is starting to look really good!

We're feeling good about this model, so we're going to let it run full-speed ahead for another day or two. If it doesn't end up converging by the end of the weekend, we're going to need to either further refine our dataset or experiment with the model weights. Expect an update in the next couple of days!

Image samples taken from the morning of the 16^th revealed that our model is definitely on the right track! The GAN still has a hard time with arms and hands, but the clothes are starting to become coherent, and the faces have more natural expressions.

At this point, we figure that because the model is on the right track, it's time to throw more processing power at it. We've upgraded our cluster to use more expensive GPUs at a considerably higher cost, but it effectively doubles our sample size and triples the rate at which we can run the simulation, which means we'll get better results faster. Stay tuned!

September 12 was the first day of training with our refined data set. Prior to the 12^th, we were training on a much broader set of data, which included multiple artwork styles (chibi, line drawings, etc.) or drawings with lots of artifacts (crazy backgrounds, weapons, etc.). This made it very difficult for the model to converge.

With the refined data set, we were able to converge on faces much quicker than in previous training runs, but the model has hardly any clue about arms, the outfits are wonky, and there are lots of random artifacts on the page. Still, it's clear that this is a huge step in the right direction!

After another couple days of training, it has became very clear that the model is not converging on bodies.

Our hypothesis is that there is too much variety in our training data, so we've set about training a classifier by manually labeling THOUSANDS of images (ourselves, by hand!) to help us identify only the highest, top-quality Waifu material from our massive repository of training data. We're going to kick off the process again with new training data and will report back soon!

After setting up our GAN and training it on a very large dataset for a week, we finally started to see images that at least somewhat resembled Anime girls. Unfortunately (or fortunately, if you're here for a laugh), the results speak for themselves.

This original model seemed to figure out (but not well) what goes on faces, and the girls have hair that mostly comes out of their head, but bodies were simply... well, they're simply hilarious is what they are. Hilarious, but not Waifu material. We're gonna let it chug for a little while longer and see if we can get over the hump.