Scouter Demo

EDIT: Minimum Viable Product

A newer version of this project is now open source and would love contributors.

I am super bored of mobile and have trouble with the last 20% - but am trying to get into the store at least. The project management page for Phase 1 has:

• the current sprint and status
• roadmap and list of tasks

Reduced functionality

Distilling the basic functionality and user experience:

Player 1:

1. Select “new thing” from dashboard
2. obtain an image (streaming, or saved)

Device 1:

1. generate clues and embeddings with machine learning
2. generate mask image
3. make available online

Player 2:

1. Select thing from dashboard
2. obtain an image (streaming, or saved)

Device 2:

1. generate clues and embeddings with machine learning
2. compare with original image
3. determine points for matches

Dashboard:

1. Action buttons
2. List of things and players.

AI

One of the best features of the new version is using AI embeddings to match images. The clues then become less challenging to make accurate matches. Text to image machine learning can use clues to generate a mask image.

All machine learning is done on-device and offline first for remote locations.

Backend

The backend has mostly been offloaded to Supabase for PostgreSQL (over MySQL with AppWrite).

• PGVector postgres extensions store machine learning vector embeddings for querying.
• Realtime functionality of tables sync changes across devices.
• Storage capabilities hold the user images
• Edge functions handle background processing.
• Authentication is for free

Multiplatform

Kotlin compose multiplatform allows targeting Android, iOS and Web.

Original Post

This is a demo for Scouter - a mobile version of “I spy with my little eye”.

How it Works

You can read the full spec doc, but the gist of it is:

0. turn based game
1. take a picture of something
2. scan it using machine learning for clues
3. another player takes a picture to guess based on clues
4. get points for guessing right

Clues

Clues include:

• dominant color of an image
• where the image was taken and geocode information
• what labels machine learning finds
• what objects machine learning finds
• what text machine learning finds
• what landmarks machine learning finds
• what rhymes with the dominant machine learning field

The basic clue is location and color.

Issues

Making mobile applications in my spare time does normally suit me as I get bored from doing it for a living.

One thing that became apparent is the need to have manual override of the clues gatherd about an image.

In the full version (not demo), there exist such functionality as well.

As usual, unless formally enforced, my test automation is lacking.

Getting Paid

The monetary incentive is in-app purchases of additional clues when guessing.

Manually overriding clues, might be a good purchasable add-on as well.

As well as unlocking limits on the amount of games you can play and specific game limits like amount of players and images.

Implementation

Once the app was spec’ed out, it took about 4 months to build a prototype.

Frontend

The prototype was written in 2016 using native android and kotlin. Some of the libraries include:

• android architecture components
  • lifecycle
  • live data
• firebase
  • realtime db
  • messaging
  • auth
• mi kit / tensorflow

Backend

The push messaging component is written on the backend to coordinate when a players turn ends and begins.

It is written in GoLang with an API and a background scheduler task.

There is also a simple HTML frontend for creating ad-hoc messages and other admin tasks.

Going Forward

Frontend

Obviously an iOS and Web solution would be preferable to have as well.

I have attempted to rewrite in React-Native and Flutter already. Made some progress but this prototype is too far ahead.

Backend

The backend has to be secured and put into my infrastructure which still needs upgrading (the final 1% of version 2 is elusive).

Firebase is pretty awesome, but I worry about the cost of scaling.

If it were cheaper to scale myself, it would be nice to get my Garden DB mobile SDK syncing nicely over MQTT (maybe one day when I am 80). But for now the free tier is enough.

Testing

Testing the validity of the automatic clue engine will be a challenging task.

It would be nice to generate a customized machine learning model for the application.

Conclusion

Overall, not a bad app and use case.

I enjoyed it, learned a bunch and think it has potential.