Sep 18

Memkite – a startup developing mobile/wearable startup Atbrox has helped foster – has created an annotated Deep Learning Bibliography, check out URL below:

http://deeplearning.university

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Tagged with:
Sep 08

Memkite – a startup developing mobile/wearable startup has helped foster – has a demo video of:
Deep Learning image recognition combined with Search for Education, check out URL below:

http://deeplearning.education

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Tagged with:
Apr 02

Memkite – a mobile/wearable search startup that Atbrox has helped foster – describes technical feasibility of building Hitchhiker’s Guide to the Galaxy, check out the blog post at:

http://blog.memkite.com/2014/04/01/technical-feasibility-of-building-hitchikers-guide-to-the-galaxy-i-e-offline-web-search-part-i/

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Aug 26

Atbrox is one of 8 European partners in a research project on cloud computing. This is a great opportunity for us to learn how and help out in making cloud computing more efficient.

Underneath is a translation of the leading partner’s – Department of Computer Science (IFI), The Faculty of Mathematics and Natural Sciences, University of Oslo – description of the project:

ENVISAGE Project
The planet’s data storage and processing is about to move up in the clouds. Sharing and rental of computing resources across geographic boundaries creates new opportunities, especially for companies who can now access the computing power they couldn’t previously afford.

Professor Einar Broch Johnsen at IFI has received financial support from the European Commission to conduct a research project to make the transition to the cloud more attractive, especially for industry. The main advantage of cloud-driven computing is to use and pay for what you need. But how a business can predict and estimate the resources used in the design phase of a project is not nearly well enough developed, which can easily lead to bad miscalculations. This will ENVISAGE try to change. ENVISAGE project has eight partners in five countries and has as main objective to facilitate the development of virtualized services. By building parts of the legal basis of the service agreement between the customer and the provider into the system, the customer / business easier to fine-tune their consumption and thereby, i.e. save time and money. Potential users for ENVISAGEs technology are companies that develop software. The technology will giving them the opportunity to improve utilization of cloud resources. The benefits of this are obvious, and being at the forefront of this development project hopes to help businesses can improve profitability significantly. ENVISAGE will run until autumn 2016 and is funded through the EU 7th Framework.

Best regards,
Amund Tveit
amund@atbrox.com

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Tagged with:
May 28

The purpose of Continuous Deployment is to increase Quality and Efficiency,
see e.g. The Software Revolution behind Linkedin’t Gushing Profits or read on

This posting presents an overview of Atbrox’ ongoing work on Automated Continuous Deployment. We develop in several languages depending on project or product, e.g. C/C++ (typically with SWIG combined with Python, or combined with Objective C), C# , Java (typically Hadoop/Mapreduce-related) and Objective-C (iOS). But most of our code is in Python (together with HTML/Javascript for frontends and APIs) and this posting will primarily show Python-centric continuous deployment with Jenkins (total flow) and also some more detail on the testing Tornado apps with Selenium.





Continuous Deployment of a Python-based Web Service / API

Many of the projects we develop involve creating a HTTP/REST or websocket API that generically said “does something with data” and has a corresponding UI in Javascript/HTML. The typical building stones of such a service is shown in the figure:

The flow is roughly as follows

  1. An Atbrox developer submits code into a git repo (e.g. Bitbucket.org or Github.com repo)
  2. Jenkins picks up the change (by notification from git or by polling)
  3. Tests are run, e.g.
    py.test -v --junitxml=result.xml --cov-report html --cov-report xml --cov .
    
    1. Traditional Python unit tests
    2. Tornado web app asynchronous tests – http://www.tornadoweb.org/en/stable/testing.html
    3. Selenium UI Tests (e.g. with PhantomJS or xvfb/pyvirtualdisplay)
    4. Various metrics, e.g. test coverage, lines of code (sloccount), code duplication (PMD) and static analysis (e.g. pylint or pychecker)
  4. If tests and metrics are ok:
    1. provision cloud virtual machines (currently AWS EC2) if needed with fabric and boto, e.g.
      fab service launch
      
    2. deploy to provisioned or existing machines with fabric and chef (solo), e.g.
      fab service deploy
      
  5. Fortunately Happy customer (and atbrox developer). Goto 1.

Example of selenium test of Tornado Web Apps with PhantomJS

Tornado is a python-based app server that supports Websocket and HTTP (it was originally developed by Bret Taylor while he was a FriendFeed). In addition to the python-based tornado apps you typically write a mix of javascript code and html templates for the frontend. The following example shows how to selenium tests for Tornado can be run:

Utility methods for starting a Tornado application and pick a port for it

import os
import tornado.ioloop
import tornado.httpserver
import multiprocessing


def create_process(port, queue, boot_function, application, name, 
                    instance_number, service, 
                    processor=multiprocessing):
    p = processor.Process(target=boot_function, 
                          args=(queue, port, 
                               application, name,
                               instance_number, service))
    p.start()
    return p

def start_application_server(queue, port, application, name, 
                             instance_number, service):
    http_server = tornado.httpserver.HTTPServer(application)
    http_server.listen(port)
    actual_port = port
    if port == 0: # special case, an available port is picked automatically
        # only pick first! (for now)
        assert len(http_server._sockets) > 0
        for s in http_server._sockets:
            actual_port = http_server._sockets[s].getsockname()[1]
            break
    pid = os.getpid()
    ppid = os.getppid()
    print "INTERNAL: actual_port = ", actual_port
    info = {"name":name, "instance_number": instance_number, 
            "port":actual_port,
            "pid":pid, 
            "ppid": ppid, 
            "service":service}
    queue.put_nowait(info)
    tornado.ioloop.IOLoop.instance().start()

Example Tornado HTTP Application Class with an HTML form

# THE TORNADO CLASS TO TEST
class MainHandler(tornado.web.RequestHandler):
    def get(self):
        html = """
<html>
<head><title>form title</title></head>
<body>
<form name="input" action="http://localhost" method="post" id="formid">
Query: <input type="text" name="query" id="myquery">
<input type="submit" value="Submit" id="mybutton">
</form>
</body>
</html>
"""
        self.write(html)

    def post(self):
        self.write("post returned")

Selenium unit test for the above Tornado class


class MainHandlerTest(unittest.TestCase):                                                                                        
    def setUp(self):                                                                                                             
        self.application = tornado.web.Application([                                                                             
            (r"/", MainHandler),                                                                                                 
            ])                                                                                                                   
                                                                                                                                 
        self.queue = multiprocessing.Queue()                                                                                                                                                                                                        
        self.server_process = create_process(0,self.queue,start_application_server,self.application,"mainapp", 123, "myservice") 
        self.driver = webdriver.PhantomJS('/usr/local/bin/phantomjs')                                                            
                                                                                                                                 
    def testFormSubmit(self):                                                                                                    
        data = self.queue.get()                                                                                                  
        URL = "http://localhost:%s" % (data['port'])                                                                             
        self.driver.get('http://localhost:%s' % (data['port']))                                                                  
        assert "form title" in self.driver.title                                                                                 
        element = self.driver.find_element_by_id("formid")      
        # since port is dynamically assigned it needs to be updated with the port in order to work                                                         
        self.driver.execute_script("document.getElementById('formid').action='http://localhost:%s'" % (data['port']))            
                                                                                                                                 
        # send click to form and receive result??                                                                                
        self.driver.find_element_by_id("myquery").send_keys("a random query")                                                    
        self.driver.find_element_by_id("mybutton").click()                                                                       
        assert 'post returned' in self.driver.page_source                                                                        
                                                                                                                                 
                                                                                                                                 
    def tearDown(self):                                                                                                          
        self.driver.quit()                                                                                                       
        self.server_process.terminate()                                                                                          
                                                                                                                                 
                                                                                                                                 
if __name__ == "__main__":                                                                                                       
    unittest.main(verbosity=2)                                                                                                   
                                                                                                                                 

Conclusion
The posting have given and overview of Atbrox’ (in-progress) Python-centric continuous deployment setup, with some more details how to do testing of Tornado web apps with Selenium. There are lots of inspirational and relatively recent articles and presentations about continuous deployment, in particular we recommend you to check out:

  1. Etsy’s slideshare about continuous deployment and delivery
  2. the Wired article about The Software Revolution Behind LinkedIn’s Gushing Profits
  3. Continuous Deployment at Quora

Please let us know if you have any comments or questions (comments to this blog post or mail to info@atbrox.com)

Best regards,
The Atbrox Team

Side note: We’re proponents and bullish of Python and it is inspirational to observe the trend that several major Internet/Mobile startups/companies are using it for their backend development, e.g. Instagram, Path, Quora, Pinterest, Reddit, Disqus, Mozilla and Dropbox. The largest python-based backends probably serve more traffic than 99.9% of the world’s web and mobile sites, and that is usually sufficient capability for most projects.

Digg This
Reddit This
Stumble Now!
Buzz This
Vote on DZone
Share on Facebook
Bookmark this on Delicious
Kick It on DotNetKicks.com
Shout it
Share on LinkedIn
Bookmark this on Technorati
Post on Twitter
Google Buzz (aka. Google Reader)
Tagged with:
preload preload preload