Welcome Page

The most simple of them all. Miles Welcome Page:

Setup Page

The Setup page walks users through the process of entering their API keys and other necessary configurations for M.I.L.E.S.

💡 Tips and Tricks

• 🤔 Ask About Capabilities: Not sure what Miles can do? Just ask him.

• 💰 Cost-Effective Mode: Find Miles too pricey? Ask him to be cheaper, and he'll switch to a more cost-effective model and system prompt. 💸

• 🎭 Personality Customization: Miles can rewrite his own personality and instructions on command! Tell him to be anything you want. 🤯

• 🔑 API Keys: You'll need a few API keys to use Miles, but the app will guide you. Only OpenAI requires payment. 💳

• ⚙️ Smart Wake Word: Miles will automatically detect when or when not to use the wake word, so if he asks a follow up question, he won't make you say 'Miles' again.

🔒 Your Privacy Matters

Here is an in-depth look at how Miles ensures your information is safe and secure:

• Local Processing: All voice commands, images captured by the webcam, and other inputs are processed locally on your device. This includes wake word detection, image recognition, and audio processing for voice commands.

• Secure Data Transmission: When data needs to be sent to the OpenAI API (for example, to fetch responses from GPT-4 Turbo or GPT-3.5 Turbo), it is securely encrypted and transmitted over a protected connection. Your API keys and any sensitive information never leave the app.

• Voice and Image Data:

  • Voice audio is processed in real-time on your device. Absolutely no voice data is sent anywhere outside your computer.
  • Images captured for recognition are encrypted in Base64 format before being securely sent to the necessary service for analysis.

• Spotify Integration: Communication with Spotify is encrypted, adhering to Spotify's security protocols. This ensures that your music preferences and commands are securely handled.

• Persistent Memory and Calculations:

  • The persistent memory feature, where you can ask Miles to remember certain information, is stored locally on your device. This data is not transmitted outside unless explicitly required for a query.
  • Calculations and the use of the built-in calculator are performed locally, with results generated without the need to communicate with external servers.

• Local Features:

  • Multi-tasking, contextual awareness, and other intelligent features of Miles are primarily processed on your device. This limits the amount of data that needs to be transmitted and ensures faster, more secure responses.
  • Internet browsing and information retrieval are done securely, with precautions to anonymize and protect your queries.

• Wake Word Detection: Utilizes short-term listening windows to ensure that continuous audio recording is not stored or transmitted. This ensures that your ambient conversations remain private and are not processed by Miles.

• Data Retention: Miles is designed to respect your privacy by not retaining data longer than necessary. For example, audio recordings are immediately discarded after processing your command, and any remembered information is deleted upon your request or when it is no longer needed.

• Local Storage and Encryption: All data stored by Miles for operational purposes, such as your preferences or the information you've asked to be remembered, is encrypted and stored locally on your device.

🗣️ Natural Language Commands

Miles is powered by a language model, so no specific commands are needed. Just speak naturally, and he'll understand! For example:

• 🎶 "Miles, play that funky tune that gets me groovin'!"
• 💃 "Miles, do I have RBF?"
• 🤖 "Miles, please store the date of my bestie's birthday so I don't forget it again!"

Let your imagination run wild with Miles!

Planned upcoming features with progress %

These are all planned features that will soon be implemented, you'll see the date last updated, and current progress as of that date.

Smart Wake Up

4-30-24 - 60% complete - lots of ideas, plans laid out

• Say anything to activate Miles
• It understands context, meaning it won't activate by you just talking
• Examples: Instead of just saying Miles, you could say "i need help" or "wake up" or "wake up miles" or hi miles, hey miles, okay miles, and miles
• Annoyance mode: Allows slower but smarter commands, you don't need a wake word at all anymore, just talk when you wanna talk, Miles will always be listening and transcribing background context and conversations to differentiate between a question, task, and if it's directed towards Miles or someone else.
• I'm highly confident I can recreate the useability of Jarvis from Ironman, where you just talk whenever you want and it works. Although it will be very expensive and very slow, probably 2-7 seconds slower.
• Update: I finished the annoyance mode, it works great and can recreate the useability of jarvis, i got it to work with local models for everything too.

Plugins

3-29-24 - 100% complete - finished

• Allows anyone to add a python function or group of python functions to Miles' tool list.
• All you have to do is write a python function and follow the correct docstring format

Home Assistant integration

3-29-24 - 100% complete - finished

• During setup, you can provide your Home Assistant tokens and choose which devices you want Miles to control.

Settings and Config Menu

3-31-24 - 10% complete - extremely hard to do, would have to change every python variable to an import

• Allows you to change API keys and preferences after setup.

Dynamic Background Learning

3-29-24 - 0% complete - rough ideas in place, still researching

• Everytime you aren't talking to Miles, he is processing, compacting, and researching your past conversations to learn how to better assist you.
• Agentic behavior behind the scenes
• Allowed to set schedules for tasks, learn common commands and at what time you ask them, and act upon them autonomously when trends are strong.

Projection UI

3-31-24 - 100% complete - finished

• Changes UI to simplistic black background with smooth animations meant for projecting onto a wall

https://github.com/small-cactus/M.I.L.E.S/assets/125771841/b3286b3c-f88b-4315-bbe5-850e8d681e9d

This is a realtime demo of the projection UI with sound (modulated to emulate jarvis, not sure if I will keep it), the black background makes it appear on the wall as if it has a transparent background

🤝 Contribute to Development

Your feedback shapes M.I.L.E.S! Though this is a beta release, I eagerly await your feature requests and issue reports for our final launch on GitHub.

Contact: anthonyhayward1000@gmail.com