atbrox

Tracer Bullet Development

Tracer Bullet Development is finding the major “moving parts” of a software system and start by writing enough code to make those parts interact in a real manner (e.g. with direct API-calls, websocket or REST-APIs), and as the system grows (with actual functionality and not just interaction) keep the “tracer ammunition” flowing through the system by changing the internal interaction APIs (only) if needed.

Motivation for Tracer Bullet Development

1. integration is the hardest word (paraphrase of an old tune)
2. prevent future integration problems (working internal APIs from the start)
3. have a working system at all times (though limited in the beginning)
4. create non-overlapping tasks for software engineers (good management)

(Check out the book: Ship it! A Practical Guide to Successful Software Projects for details about this method)

Examples of Distributed Tracer Bullet Development

Let us assume you got a team of 10 excellent software engineers who had never worked together before, and simultaneously the task of creating an first version and working distributed (backend) system within a short period of time?

How would you solve the project and efficiently utilize all the developers? (i.e. no time for meet&greet offsite)

Splitting the work properly with tracer bullet development could be a start, let’s look at how it could be done for a few examples:

1. Massively Multiplayer Online Games
Massively Multiplayer Online Games, e.g. Zynga’s Farmville, Linden Lab’s Second Life, and BioWare/LucasArt’s Star Wars Old Republic – are complex distributed systems. So what can a high-level tracer bullet architecture for such a game look like? Services that might be needed are:

1. GameWorldService – to deal with the game world, assuming basic function is returning a graphic tile for a position x, y, z
2. GameArtifactService – to deal with state of various “things” in the world (e.g. weapons/utilities), e.g. growth of plants.
3. GameEconomyService – to deal with overall in-game economy and trade
4. AvatarService – to deal with player avatars and non-player characters (monsters/bots) (i.e. active entities that operate in the GameWorldService and can alter the GameArtifactService)
5. LogService – to log what happens in the game
6. StatService – calculates/monitors various statistics about the game
7. AIService – e.g. used by non-player characters for reasoning
8. UserService – to deal with users (profiles, login/passwords etc, metainfo++)
9. GameStateService – overall game state
10. ChatService – for interaction between players
11. ClientService – to deal with various software clients users use, e.g. ipad client, pc client
12. CheatMalwareDetectionService – always someone looking to exploit a game
    UserService (to deal with state/metainfo regarding the user),

Already more services (12) than software engineers (10), but let us create a beginning of a draft of at tracer bullet definition in a json-like manner.

tracerbullets = {
"GameWorldService":{
  "dependencies":["GameStateService","LogService"],
  "defaultresponse":{"tiledata_as_json_base_64": ".."},
  "loadbalancedserveraddress":"gameworld.gamecomp.com"},

"GameArtifactService":{
  "dependencies":["GameStateService","GameWorldService"],
  "defaultresponse":{"artifactinfo": ".."}
  "loadbalancedserveraddress":"gameartifacts.gamecomp.com"},

"AvatarService":{
},

"GameEconomyService":{
}
"
}

Atbrox’ (internal) Tracer Bullet Development Tool – BabelShark
The game example resembles RPC (e.g. Avro) and various deployment type definitions (e.g. Chef, or Puppet) but it focused on specifying enough information (but not more) to get the entire (empty, with default responses) system up and running with it’s approriate host names (which can be run on one machine for testing with either minor /etc/hostname file changes or running a local dns server). When the system is running each request appends the received default responses to its default response so one can trace the path of e.g. REST/HTTP or websocket calls through the system (e.g. if a call to the GameWorldService uses both GameStateService and LogService as below, this will be shown in the resulting json from GameWorldService). When the (mock-like) default responses are gradually being replaced with real services they can be run as before, and when they are properly deployed just removing the DNS entry in /etc/hosts or the local dns server to get real data. Proxying external services (e.g. Amazon Web Services) can be done in a similar manner. This can in overall make it easier to bridge development situation with deployment situation.

In Atbrox we have an internal tool called BabelShark that takes an input tracer bullet definition (json) and creates Python-based tracer bullet system code (using Bret Taylor’s Tornado websocket support) and also creates corresponding websocket commandline client and javascript/html clients for ease of testing all components in the system. Technically it spawns one tornado process per service (or per instance of a service if more than one), dynamically finds available port numbers and communicates them back to , creates a new /etc/hosts file with the requested host names per service (all pointing to localhost), and a kill-shell-script file (note: you quickly get a lot of processes this way, so even if the multicores are humming you can quickly overflow them, so nice to be able to kill them).

Example 2. Search Query Handling
The prerequisite for search is the query, and a key task is to (quickly) understand the user’s intention with the query (before actually doing anything with the query, such as looking up results in an index).

A few questions needs to be answered about the query: 1) what is the language of query?, 2) is the query spelled correctly in the given language? 3) what is the meaning of the query? 4) does the query have ambiguous meaning (either wrt language or interpretation), 5) what is the most likely meaning among the most ambiguous ones? So how can a tracer-bullet development for this look like?

tracerbullets = {
"LanguageDeterminator":{
  "dependencies":["KlingonClassifier", "EnglishClassifier"],
  "defaultresponse":{"sortedlanguagesprobabilities":[{1.0:"English"}]}
},

"SpellingIsCorrect":{
   "dependencies":["LanguageDeterminator","KlingonSpellChecker"],
   "defaultresponse":{"isitspelledcorrectly":"yes"}
},

"MeaningDetermination":{
   "dependencies":["LanguageDeterminator", "NameEntityDeterminator"],
   "defaultresponse":{"meaning":"just a string with no entities"},
},

"Disambiguator": {
   "dependencies":["MeaningDetermination", ".."],
   // specialized for the query: Turkey - is it about the country or 
   // or about food (i.e. right before thanksgiving)
   "defaultresponse":{
    "disambiguatedprobabitity":[{0.9:"country"},{0.1:"bird"}]
}
}



}

Conclusion

Have given an overview of tracer bullet development for a couple of distributed system cases, and have also mentioned how our (internal) tool supports Distributed Tracer Bullet Development.

If you are keen to learn more and work with us here in Atbrox, please check out our jobs page. Atbrox is a bootstrapped startup working on big data (e.g. hadoop and mapreduce) and search (we also work with and own parts of a few other tech startups).

Best regards,

Amund Tveit (@atveit)
Atbrox (@atbrox)