Marc's Notes

Last week, I joined machine learning enthusiasts from around the world in Prague for Europe's biggest conference on ML and AI applications, ML Prague 2023.

The event featured a broad lineup of speakers, covering a wide range of topics - from the latest breakthroughs in natural language processing to cutting-edge computer vision applications. In this blog post, I want to write about some of the most interesting talks from the conference and highlight the key takeaways. So, without further ado, let's dive in and start with two topics from the area of image processing.

Image Processing

3D Pose Estimation in Sport

Piotr Skalski from Roboflow showed us how he was able to reproduce parts of the Video Assistant Referee (VAR) system shown in the following clip.

He used YOLOv7 and two cameras that record from different perspectives to track a skeleton of his body.

When using this technique, there are some key points to consider:

• Use at least two cameras with the same configuration
  • For best results, position them above the playing field to reduce player occlusions
  • For better results, use more than two cameras
• Avoid or remove any lens distortion before processing the video recordings.

Take a look at his GitHub repository to see the results.

Multi-Model Machine Learning based Industrial Vision Tool for Assembly Part Quality Control

Aimira Baitieva from Valeo offered insights into anomaly detection in images using neural networks.

One interesting idea was to train a model that outputs embeddings for images, which represent their information well enough. You could then create a "good" embedding for normal parts (without anomalies) to compare against. If an embedded image has a dissimilarity above a certain threshold when compared with the good embedding, one can mark it as an anomaly.

Their final multi-model approach, however, consisted of three steps:

1. Create a segmented anomaly map:
   1. Segment the image to get a segmentation map
   2. Use an anomaly detector to get an anomaly map
   3. Combine both maps
2. Extract features from this map
3. Classify the image based on the features.

This turned out to work quite well and is currently in use.

Large Language Models

LLM-driven Game Characters

Marek Rosa from GoodAI shared their progress in creating LLM-driven game characters.

In their AI Game, characters use a large language model to generate their thoughts, actions and speech. The player can interact with them via a chat interface and with normal in-game actions like exchanging objects.

Every character has a long-term memory which has been implemented via a vector database. New thoughts and interactions are preprocessed and then saved into the database. When needed, the database will be queried with respect to

• recency
• importance
• and relevance.

The short-term memory is controlled with the current context of the language model.

This combination of short and long-term memory lets agents learn continually by using it to plan actions based on past observations, experience, its current environment and thoughts.

Personally, I think that LLMs will completely redefine the role of NPCs (Non-Player Characters) in video games. Today, the player-controlled character is often the central figure in the game's story while other characters are less important. In the near future, the player will take on a more supporting role and the game does not have a clearly defined protagonist. The story will be open-ended and allows the players and NPCs to make choices that affect the outcome and direction of it. More games will adopt a sandbox-style approach, similar to Minecraft, where there is no apparent goal or ending of the game.

Other interesting projects that try to use LLMs to create autonomous agents are

• AutoGPT
• Voyager Minecraft Agent

Conclusion

In conclusion, the ML Prague Conference 2023 was a fantastic opportunity for ML professionals to come together, learn about the latest trends and innovations in the field, and connect with others. I had a great time and am glad that I was able to be there. I hope you enjoyed this short recap of the conference. If you've been there too, let's connect on LinkedIn and have a chat about your favorite talks.