Richard Nelson @imwabz - Tumblr Blog

Posts

On the 29th of March, the Australian Government launched an iOS application on the AppStore. It’s a pretty neat little app with lots of…

Details of an iOS app which recognises license plates and does registration and insurance lookups.

#ComputerVision

Augmented Reality Flight Status on iOS

At WWDC2018, Apple announced and released an addition to their family of AR technologies: image recognition. This feature allows an iOS device to detect a given image within a scene, and track it in 3 dimensional space in real time. They gave some impressive demonstrations, and I had an idea to try to overlay an image of the world with little aeroplanes, denoting where real world aircraft are in realtime. Here’s a video of the end result, and the rest of this post will contain some implementation details.

As you can see in the video, the tracking of the image works remarkably well. Even when the physical image is shaken rather violently, ARKit has no trouble tracking the positioning of the image in 3d space. Also impressive is the ability to maintain tracking when the image is only partly in view.

Once you enable image detection in ARKit and load a reference image, iOS will add an anchor for you which tracks around as the camera and image moves. The reference image must be pre-defined, have a fairly evenly spread histogram, and you must let Xcode know of the physical image dimensions as accurately as possible.

Once image detection is set up, we need to overlay the little aeroplane images. The data source used provides the latitude, longitude, altitude and heading of the aircraft. All of these values are used when placing the image. The altitude was originally ignored, but because so many aircraft end up at similar points (especially around Sydney) and the images are not to scale, the image positions would conflict and a strange shimmering effect happened which looked wrong. So the real altitudes are used as a slight height offset, which fixed the shimmering and also happened to increase the realism.

To convert latitude and longitude to an x,y coordinate in 2d space requires some simple trigonometry. But this also requires the correct type of map projection: equidistant cylindrical projection. This type of projection conserves distances along the meridian, making lat/lon to x/y mapping accurate:

Then the heading is used to rotate the plane (side node: aeroplanes on geometry planes can be incredibly confusing nomenclature) around the Y-axis to show which direction the aircraft is going. The first image used when developing was not of the correct projection type, making the placements incorrect - it was incredibly satisfying to see the aircraft placed correctly over landmarks once the correct map projection was used!

The code to place each aeroplane looks like this:

func loadAeroplanes(image: ARReferenceImage) { AeroplaneDataSource.getRealtimePositioning { aeroplanes in aeroplanes.forEach { aeroplane in guard let lat = aeroplane.Lat, let lon = aeroplane.Long, let call = aeroplane.Call else { return } let material = SCNMaterial() material.diffuse.contents = UIImage(named: "plane") let geometry = SCNPlane(width: 0.01, height: 0.01) geometry.materials = [material] let planeNode = AeroplaneNode(geometry: geometry) planeNode.originalColour = material.diffuse.contents as? UIColor let x = Float(image.physicalSize.width / 360 * CGFloat(180 + lon)) - Float(image.physicalSize.width / 2) let z = Float(image.physicalSize.height / 180 * CGFloat(90 - lat)) - Float(image.physicalSize.height / 2) var y: Float = 0.005 if let altitude = aeroplane.Alt { y = Float(altitude) * 0.000000025 } planeNode.simdPosition = float3(x: x, y: y, z: z) planeNode.eulerAngles.x = -.pi / 2 if let heading = aeroplane.Heading { planeNode.eulerAngles.y = Float(heading) * Float.pi / 180 } planeNode.flightData = aeroplane self.imageAnchorNode?.addChildNode(planeNode) } } }

Then, to create a HUD on top of the aircraft when the user types a flight number in, we need to:

Find the appropriate plane node in our set of planes

Display the hud view controller on top (and slightly behind) the relevant node

Interestingly, although it’s undocumented, you can use a UIView as the plane’s material’s diffuse contents. This allows you to create a layout in storyboards, and display it on a plane in your scene. Here’s a screenshot of the storyboard layout used in this example:

And the code to attach this view to a SCNPlane:

let plane = SCNPlane(width: 0.16, height: 0.1015) let view = self.hudViewController.view.subviews.first view?.layer.cornerRadius = 40 view?.layer.borderColor = UIColor(red: 0.7, green: 0.0, blue: 0.0, alpha: 0.8).cgColor view?.layer.borderWidth = 2 plane.firstMaterial?.diffuse.contents = view plane.firstMaterial?.isDoubleSided = true //required if you place the camera behind the view self.hudNode = SCNNode(geometry: plane) self.hudNode?.eulerAngles.x = -.pi / 4 self.hudNode?.opacity = 0

Then a UIViewController is used to configure the text on the UILabels. We also iterate through all the non-matching nodes and decrease their opacities so that the searched flight number stands out. The position of the hud node is updated to hover in the correct space above the aircraft:

statusViewController.flightNumberHandler = { number in if number == "" { self.hudNode?.opacity = 0.0 self.imageAnchorNode?.childNodes.forEach { node in node.opacity = 1.0 } return } self.imageAnchorNode?.childNodes.forEach { node in guard let planeNode = node as? AeroplaneNode else { return } if (planeNode.flightData?.Call?.lowercased() == number.lowercased()) { node.opacity = 1.0 AeroplaneDataSource.getFlightStatus(flightNumber: number.lowercased().replacingOccurrences(of: "qf", with: "")) { flightStatus in DispatchQueue.main.async { self.hudViewController.configureFrom(flightStatus: flightStatus) self.hudViewController.update(altitude: planeNode.flightData?.Alt, speed: planeNode.flightData?.Speed) self.hudNode?.simdPosition = float3(x: node.simdPosition.x, y: node.simdPosition.y + 0.01, z: node.simdPosition.z - 0.08) self.hudNode?.opacity = 1.0 } } } else { node.opacity = 0.2 } } }

And voila! While the data source and ARKit boilerplate are left as an exercise to the reader, hopefully this gives some insight into how computer generated images can be mapped on to real world images and objects.

#Apple ARKit

Purchase request approved #newtoy #thankyouwifey 📷

#thankyouwifey #newtoy

My wife found me a new hobby 🐩 and now I am the new GOD to the kids. Better learn how to make at least one more animal before they ask. 🦒🦍🐖🐅🐓🐿🦓🐇🐘

YQL+ on Amazon Serverless

Yahoo has quietly open sourced the YQL+ engine, parser and SDK. It’s a powerful platform for querying and orchestrating data, and if you’re familiar with their YQL platform some concepts will be familiar. But in YQL+, your programs live in your application, not as part of a request, and you can interact with it in a more RESTful (or any) manner.

YQL+ implements a query language for services. A YQL “program” looks a little like SQL, but its sources can provide data from anywhere. For example, you could have a source that pulls data from a webservice, or redis, or MySQL, or DynamoDB. Your program may query multiple sources, join them, have the output of one as the input of another, or a complicated combination. The YQL+ engine will figure out which sources it can execute in parallel and then provide the result(s).

One use case I can imagine is using YQL+ application as the public interface for a mobile app. It’s good at orchestrating data, querying multiple downstream sources, joining data, perhaps executing a little business logic the apps have no business knowing about themselves and returning only the necessary data.

Here, I have combined YQL+ with Amazon Serverless to create microservices. Each endpoint is its own lambda function. The source for this example is located at https://github.com/wabzqem/yql-plus-serverless-example

It works as follows:

Client request -> Amazon API Gateway -> lambda handler which executes appropriate YQL Program -> downstream source(s) -> back to client

The files in the “function” package are the ones mapped to each AWS lambda function. The files in the “sources” package are the YQL+ Source classes, which fetch/manipulate downstream data. Table names are mapped to Sources in SourceModule.java. And the methods in the source classes are annotated with @Query (more on that later).

This example has the following endpoints:

GET / - fetches the current stocks POST / - adds a stock to the portfolio and fetches data for it GET /update/ updates the current prices in the portfolio

See README.md in the repository for deployment instructions.

To get an idea of what’s going on here, let’s see how the update endpoint works. Here’s the YQL program which is mapped to the /update/ endpoint (see net.whatsbeef.portfolio.webservice.function.UpdateStock to see how this gets mapped):

PROGRAM(); CREATE TEMPORARY TABLE updated AS (SELECT * FROM stocks); SELECT * FROM updateStocks(@updated) OUTPUT AS result;

This program has no inputs. First, a temporary table is created, which is the output of the stocks query. The socks “table” is mapped to the StockSource class, and no input is given. The StockSource query method without any parameters simply fetches all stocks from the database.

A critical point here is that the output of this query is passed as input to the updateStocks table. When doing this, the YQL engine won’t execute these statements in parallel (obviously, but it does otherwise) and the types are mapped appropriately. The updateStocks table’s source does several HTTP requests to update the stock data, updates DynamoDB and returns the results.

Let’s have a look at the putStock program:

PROGRAM( @stockId string, @quantity int64, @boughtPrice double ); CREATE TEMPORARY TABLE putTable AS (SELECT * FROM putStock(@stockId, @boughtPrice, @quantity)); SELECT * FROM updateStocks WHERE stockId = @putTable[0].stockId OUTPUT AS result;

This program takes 3 inputs which are pretty self-explanatory. YQL maps int64 to java’s Long type. YQL+ chooses which @Query method in your source based on the number of parameters - not the types unfortunately. But the types must match correctly.

The second SELECT statement takes the first result from the temporary putTable query, and only the given stock is updated.

The return value of a Source is generally your model object. I’ve used simple kotlin data classes, and they work fine getting passed from one source to another.

And as mentioned, requests/responses are simply standard REST requests. Here's a sample GET:

$ curl -s https://<hidden>.execute-api.ap-southeast-2.amazonaws.com/Prod/ | python -m json.tool [ { "boughtDate": 1506928655152, "boughtPrice": 34.45, "currentPrice": 4.33, "quantity": 3000, "stockId": "\u20180354.HK'", "stockName": "CHINASOFT INT'L" }, { "boughtDate": 1506928693550, "boughtPrice": 34.45, "currentPrice": 59.95, "quantity": 3000, "stockId": "2318.HK", "stockName": "PING AN" } ]

#yahoo #yql-plus

ARKit + SceneKit + Livestreams

This is a brief writeup of my last hackday demo at Yahoo7 :)

At WWDC17, Apple introduced ARKit which makes it very easy to create an augmented reality experience which runs surprisingly well on Apple’s latest hardware. In this demo, there are 4 livestreams running simultaneously, along with a simple particle emitter:

When the screen is tapped, we first detect if an existing node (video) has been tapped with a hit test. If it is, we set the volume on the tapped video and mute all the others:

// If a node is tapped, set its volume and silence the rest let location = touches.first?.location(in: self.sceneView) let hits = self.sceneView.hitTest(location!, options: [SCNHitTestOption.ignoreHiddenNodes: SCNHitTestOption.ignoreChildNodes]) if let tapped = hits.first?.node { if let player = nodePlayerMap[tapped] { nodePlayerMap.forEach { (arg) in let (_, myPlayer) = arg myPlayer.volume = 0.0 } player.volume = 1.0 } return }

An AVPlayer is not a 3d object that can be placed, so we create a SpriteKit scene which is attached to the anchor. The SKScene has its geometry set, using the SKVideoNode (initialised with an AVPlayer) as a texture/material to be placed on the plane:

let videoSKNode = SKVideoNode.init(avPlayer: player); videoSKNode.size = spriteKitScene.size videoSKNode.play() videoSKNode.yScale = -1; videoSKNode.position = CGPoint(x: spriteKitScene.size.width / 2.0, y: spriteKitScene.size.height / 2.0) spriteKitScene.addChild(videoSKNode) videoNode.geometry = SCNPlane(width: 1.280 / 2, height: 0.720 / 2) videoNode.geometry?.firstMaterial?.diffuse.contents = spriteKitScene videoNode.geometry?.firstMaterial?.isDoubleSided = true

The video players are placed 1 metre in front of the camera using the same transform that the camera has. So if you face down and place the player, the screen will still be correctly facing the camera. This code is derived from the sample ARKit scene with Xcode 9:

// Create a transform with a translation of 1.5 meters in front of the camera var translation = matrix_identity_float4x4 translation.columns.3.z = -1.0 let transform = simd_mul(currentFrame.camera.transform, translation) // Add a new anchor to the session let anchor = ARAnchor(transform: transform) sceneView.session.add(anchor: anchor)

And that’s pretty much it.

#wwdc17 #apple arkit

yahooeng

Open Sourcing Screwdriver, Yahoo’s Continuous Delivery Build System for Dynamic Infrastructure

By James Collins, Sr. Director, Developer Platforms and Services, and St. John Johnson, Principal Engineer

Continuous Delivery enables software development teams to move faster and adapt to users’ needs quicker by reducing the inherent friction associated with releasing software changes. Yahoo’s engineering has modernized as it has embraced Continuous Delivery as a strategy for improving product quality and engineering agility. All our active products deliver from commit to production with full automation and this has greatly improved Yahoo’s ability to deliver products.

Part of what enabled Yahoo to make Continuous Delivery at scale a reality was our improved build and release tooling. Now, we are open sourcing an adaptation of our code as Screwdriver.cd, a new streamlined build system designed to enable Continuous Delivery to production at scale for dynamic infrastructure.

Some of the key design features of Screwdriver have helped Yahoo achieve Continuous Delivery at scale. At a high level these are:

Making deployment pipelines easy

Optimizing for trunk development

Making rolling back easy

Easy deployment pipelines: Deployment pipelines that continuously test, integrate, and deploy code to production greatly reduce the risk of errors and reduce the time to get feedback to developers. The challenge for many groups had been that pipelines were cumbersome to setup and maintain. We designed a solution that made pipelines easy to configure and completely self-service for any developer. By managing the pipeline configuration in the code repository Screwdriver allows developers to configure pipelines in a manner familiar to them, and as a bonus, to easily code review pipeline changes too.

Trunk development: Internally, we encourage workflows where the trunk is always shippable. Our teams use a modified GitHub flow for their workflows. Pull Requests (PRs) are the entry point for running tests and ensuring code that entered the repository has been sufficiently tested. Insisting on formal PRs also improves the quality of our code reviews.

To ensure trunks are shippable, we enable functional testing of code in the PRs. Internally, this is a configuration baked into pipelines that dynamically allocates compute resources, deploys the code, and runs tests. These tests include web testing using tools like Selenium. These dynamically-allocated resources are also available for a period after the PR build, allowing engineers to interact with the system and review visual aspects of their changes.

Easy rollbacks: To allow for easy code rollbacks, we allow phases of the pipeline to be re-run at a previously-saved state. We leverage features in our PaaS to handle the deployment, but we store and pass metadata to enable us to re-run from a specific git SHA with the same deployment data. This allows us to roll back to a previous state in production. This design makes rolling back as easy as selecting a version from a dropdown menu and clicking “deploy.” Anyone with write access to the project can make this change. This helped us move teams to a DevOps model where developers were responsible for the production state.

The successful growth of Screwdriver over the past 5 years at Yahoo has today led to Screwdriver being synonymous with Continuous Delivery within the company. Screwdriver handles over 25,000+ builds per day and 12,000+ daily git commits as a single shared entrypoint for Yahoo. It supports multiple languages and handles both virtual machine and container-based builds and deployment.

Screwdriver.cd’s architecture is comprised of four main components: a frontend for serving content to the user, a stateless API that orchestrates between user interactions and build operations, the execution engines (Docker Swarm, Kubernetes, etc.) that checkout source code and execute in containers, and the launcher that executes and monitors commands inside the container.

The diagram below shows this architecture overlaid with a typical developer flow.

To give some context around our execution engines, internal Screwdriver started as an abstraction layer on top of Jenkins and used Docker to provide isolation, common build containers, etc. We used features provided by Jenkins plugins to leverage existing work around coverage and test reports. However, as Screwdriver usage continued to climb, it outgrew a single Jenkins cluster. So in order to grow to our needs, we added capabilities in Screwdriver that allowed us to scale horizontally while also adding capabilities to schedule pipelines across a number of Jenkins clusters. As we scaled Screwdriver, we used less from Jenkins and built more supporting services utilizing our cloud infrastructure. The open-source version is focused on Kubernetes and Docker Swarm as our primary supported execution engines.

In the coming months we will expand our offering to match many of the features we have internally, including:

Mechanism to store structured build data for later use (last deployed version, test coverage, etc.)

Built-in metric collecting

System-wide templates to enable getting started quickly

Log analysis to provide insights to developers

Please join us on the path to making Continuous Delivery easy. Visit http://screwdriver.cd to get started.

yahoopolicy

Yahoo demands transparency from the Director of National Intelligence

In a letter today to James Clapper, Director of National Intelligence (DNI), Yahoo is formally urging that the U.S. government provide its citizens with clarification around national security orders they issue to internet companies to obtain user data.

While the letter makes specific reference to recent allegations against Yahoo, it is intended to set a stronger precedent of transparency for our users and all citizens who could be affected by government requests for user data. As we’ve said before, recent press reports have been misleading; the mail scanning described in the article does not exist on our systems.

DNI’s mission statement refers to “responsible and secure information sharing.” We therefore trust that the U.S. government recognizes the importance of clarifying the record in this case. On behalf of Yahoo and our global community of users, we request that the Office of the Director of National Intelligence expeditiously clarify this matter.

yahooeng

Open-sourcing Pulsar, Pub-sub Messaging at Scale

By Joe Francis and Matteo Merli, Yahoo Platforms

Pub-sub messaging is a very common design pattern that is increasingly found in distributed systems powering Internet applications. These applications provide real-time services, and need publish-latencies of 5ms on average and no more than 15ms at the 99th percentile. At Internet scale, these applications require a messaging system with ordering, strong durability, and delivery guarantees. In order to handle the “five 9’s” durability requirements of a production environment, the messages have to be committed on multiple disks or nodes.

At the time we started, we could not find any existing open-source messaging solution that could provide the scale, performance, and features Yahoo required to provide messaging as a hosted service, supporting a million topics. So we set out to build Pulsar as a general messaging solution, that also addresses these specific requirements.

Pulsar is a highly scalable, low latency pub-sub messaging system running on commodity hardware. It provides simple pub-sub messaging semantics over topics, guaranteed at-least-once delivery of messages, automatic cursor management for subscribers, and cross-datacenter replication.

Using Pulsar, one can set up a centrally-managed cluster to provide pub-sub messaging as a service; applications can be onboarded as tenants. Pulsar is horizontally scalable; the number of topics, messages processed, throughput, and storage capacity can be expanded by adding servers to the pool.

Pulsar has a robust set of APIs to manage the service, namely, account management activities like provisioning users, allocating capacity, accounting usage, and monitoring the service. Tenants can administer, manage, and monitor their own domains via APIs. Pulsar also provides security via a pluggable authentication scheme, and access control features that let tenants manage access to their data.

Application development using Pulsar is easy due to the simple messaging model and API. Pulsar includes a client library that encapsulates the messaging protocol; complex functions like service discovery, as well as connection establishment and recovery, are handled internally by the library.

Architecture

At a high level, a Pulsar instance is composed of multiple clusters, typically residing in different geographical regions. A Pulsar cluster is composed of a set of Brokers and BookKeepers (bookies), plus ZooKeeper ensembles for coordination and configuration management.

A Pulsar broker serves topics. Each topic is assigned to a broker, and a broker serves thousands of topics. The broker accepts messages from writers, commits them to a durable store, and dispatches them to readers. The broker also serves admin requests. It has no durable state. The broker has built-in optimizations; for example, it caches the data in order to avoid additional disk reads when dispatching messages to clients as well as replication clusters. Pulsar brokers also manage the replicators, which asynchronously push messages published in the local cluster to remote clusters.

Apache BookKeeper is the building block for Pulsar’s durable storage. BookKeeper is a distributed write-ahead log system, a top-level Apache project that was originally developed at and open-sourced by Yahoo in 2011. BookKeeper has an active developer community with contributors across the industry. Using the BookKeeper built-in semantics, Pulsar creates multiple independent logs, called ledgers, and uses them for durable message storage. Bookkeeper hosts, called bookies, are designed to handle thousands of ledgers with concurrent reads and writes. BookKeeper is horizontally scalable in capacity and throughput; from an operational perspective we can elastically add more bookies to a Pulsar cluster to increase capacity.

By using separate physical disks (one for journal and another for general storage), bookies are able to isolate the effects of read operations from impacting the latency of ongoing write operations, and vice-versa. Since read and write paths are decoupled, spikes in reads – which commonly occur when readers drain backlog to catch up – do not impact publish latencies in Pulsar. This sets Pulsar apart from other commonly-used messaging systems.

Managed Ledger represents the storage layer for a single topic. It is the abstraction of a stream of messages, with a single writer, and multiple readers, each with its own associated cursor position, the offset of the reader in the message stream. A single managed ledger uses multiple BookKeeper ledgers to store the data. Cursor positions are maintained in per-cursor ledgers.

A Pulsar cluster runs a ZooKeeper (another top-level Apache project open-sourced by Yahoo in 2008) ensemble used for coordinating assignment of topics among brokers, and storing BookKeeper metadata. In addition, Pulsar runs a Global ZooKeeper ensemble to store the provisioning and configuration data. At Yahoo, we have presence in multiple regions and our users create global topics that are replicated between these regions. The Global Zookeeper ensemble keeps provisioning and configuration data consistent globally. We can tolerate higher write latencies on these writes (e.g.: ~150ms latency for configuration writes).

The load balancer is a distributed service that runs on the brokers, to make sure the traffic is equally spread across all available brokers. Since Pulsar brokers have no durable state, topics can be redistributed within seconds.

Messaging Model

The Pulsar topic is the core of the system; applications and components communicate by publishing to and consuming from the same topic. Topics are created dynamically as needed when a producer (writer) starts publishing on it; and topics are removed when not in use.

Subscriptions are created automatically when a consumer (reader) subscribes to the topic. A subscription persists until it is deleted, and receives all messages published during its lifetime. Common messaging semantics (like JMS Topic or Queue) are available as subscription modes; an exclusive subscription is equivalent to a “topic,” and a shared subscription is equivalent to a “queue.”

Performance

Pulsar is designed for low-publish latencies at scale. Our typical publish latencies on average are well below 5ms. With SSD as the bookie journal device, Pulsar can achieve 99 percentile latencies of 5ms with two guaranteed copies and total ordering.

The latency remains within the acceptable range until the throughput reaches the limit of the disk IO capacity.

Pulsar supports partitioned topics, which can further increase the per-topic throughput.

Pulsar at Yahoo

Pulsar backs major Yahoo applications like Mail, Finance, Sports, Gemini Ads, and Sherpa, Yahoo’s distributed key-value service.

We deployed our first Pulsar instance in Q2 2015. Pulsar use has rapidly grown since then, and as of today, Yahoo runs Pulsar at scale.

Deployed globally, in 10+ data-centers, with full mesh replication capability

Greater than 100 billion messages/day published

More than 1.4 million topics

Average publish latency across the service of less than 5 ms

As Pulsar use grows at Yahoo, we have been scaling the service horizontally. Most of the challenges we faced were with JVM GC impacting publish latencies, and reducing failover times when the number of topics on a broker went up to tens of 1000s (now 40,000). This led to significant changes to the Pulsar broker and to BookKeeper.

Looking to the Future

We are actively engaged in pushing the scale and reliability boundaries of Pulsar further. Current improvements being worked on include:

Migrate topic between brokers in under 1 sec, from 10 sec

Improve 99.9%ile publish latencies to 5ms

Provide additional language bindings for Pulsar

Conclusion

Pulsar is a highly scalable pub-sub messaging system, production-ready and battled tested at Yahoo. We are glad to make Pulsar available as open source under Apache License Version 2.0. Detailed instructions and documentation are available at Yahoo’s Github repository. Our goal is to make Pulsar widely used and well integrated with other large-scale open source software, and we welcome contributions from the community to make that happen.

Additional Links

Pulsar Github project page: https://github.com/yahoo/pulsar

Pulsar BookKeeper enhancements: https://github.com/yahoo/bookkeeper/tree/yahoo-4.3

imwabz

NICE

ABS: Assume breached

A group recently released a set of exploits that the NSA probably just left in a place that it shouldn’t have been left in. These exploits are at least 3 years old, which means the NSA has been using them for at least that long without reporting them to the (US) companies that produce the products they exploit against. We also know that the NSA uses these kinds of tools to obtain (an obscene amount of) data about their own citizens. It also means that some group that isn’t the NSA (at this point it’s widely speculated to be the Russians) has also had these tools available to them for at least that long.

We KNOW that nation state actors are exploiting systems to get access to data without warrants. And they must also know that others are doing the same, but do nothing to get these exploits fixed.

This is exactly why large tech companies go to extreme lengths to protect their users’ data. For example, the solution that Apple has come up with to store your passwords in the cloud begins with an assumption that Apple’s own infrastructure cannot be trusted. Apple gets it.

This isn’t tinfoil hat thinking, this is actually happening.

If the ABS cannot even assume that they will be DDoS’d, how can we possibly trust them to store such a wealth of personal information securely? Nowadays, you HAVE to assume that people can get at your data. Why store names and addresses against all this information if it’s not absolutely necessary, and then show the world how inept you are?

#infosec #auspol

razzledazzle

careersatoath

An Apple TV case study: How we stayed ahead of the curve, and what we learnt along the way

Earlier this year when Apple shared our PLUS7 app as an example of excellence to a global audience in their keynote address, it was the culmination of an engaging journey for the engineering team.

When Apple announced the 4th generation Apple TV, we knew it would be a perfect fit for expanding our PLUS7 offering. The Apple TV’s simple yet elegant experience is great for designing in a way that allows users to get straight to the content they want.

Building upon our knowledge of iOS and existing Apple frameworks, the process to develop a tvOS app was fast and really enabled our engineers to push the envelope in terms of redefining the PLUS7 viewing experience on a larger screen.

Swift

With Swift still evolving fairly heavily, choosing to go with it is not without some risk and overhead. On evaluation, we decided that it was mature enough to use in a production application and that it would be the way forward for us. Although we could have reused Objective-C code from our iOS implementation, we further decided to start from scratch and our own code in the tvOS build is 100% Swift. We now believe this decision was a good one; even our test builds were remarkably stable and there are much fewer lines of code than its iOS counterpart. Additionally, we can now use our arguably better Swift code in future versions of our iOS releases.

The Build

We initially had a prototype version of the app which was built in spare time and had no input from the design team. Ultimately, it was this version that was iterated on as the build progressed. One of the main issues we encountered was 3rd party SDKs not being tvOS or bitcode compatible. With a bit of prodding, we were able to resolve most of these, with the large exception of the SDK we use for video advertising.

Advertising

On iOS and Android we use Google’s IMA SDK for video advertising, which is not compatible with tvOS at this time. Obviously, launching PLUS7 for Apple TV without video advertising is not a viable strategy for our business. So we decided to implement the VAST spec ourselves, in Swift. We had a look at some (old) open source VAST implementations which unfortunately turned out not to be suitable. At first glance a VAST implementation looks fairly straightforward.

However, it turns out that it’s much more complicated than at first glance. A video contains “cuepoints”, which indicate when advertising slots are to be played. There are three types of ad sets; pre-rolls (played before a stream), mid-rolls (several per video, in the middle of the stream) and post-rolls (played after the stream). Each cuepoint maps to a “pod” of ads. A pod is made up of a number of ads. So the first preroll might contain one 15 second ad. The first mid-roll pod may contain 4 ads of various lengths (between 15 and 60 seconds), the next one 5 ads of various lengths and so on.

When the video starts, we make one query which returns each element in each pod. However, each ad is a recursive ad call to an ad server which you have to keep querying until you get an actual creative to play. The first ad server may not have the content, but will defer to another ad server, and so on. Additionally, each ad server that is traversed through will add its own tracking URLs for each event that may occur during playback, for example each quartile that is played, play, pause, stop etc events. Further, it’s possible that following a particular path down ad servers may result in an infinite loop (!), or a playback error, in which case specific rules apply.

It is vital that the implementation of the VAST specification behaves correctly, as our business revenue depends on it. Most of the time spent on development for our 1.0 release went into this effort.

Video player

Playing content along with video ads (which are separate video streams) while providing a seamless experience to the users is not that straightforward. Each video ad is fetched ahead of time, stored in a list and passed to the video controller provided by tvOS API. The content is paused and resumed after each ad sequence in a different controller. Seeking forward over one or more “cuepoints” means that we have to play the last set of ads in a pod and so on: the user might get to the post rolls too if they are patient enough.

As much as we can, we fetch required data ahead of time. As soon as we have enough information to play a preroll, we do, and then go and do the rest of the requests for specific ads so that users aren’t waiting for this to happen every time an ad has to play.

The Apple TV experience has some unique features due to the nature of the remote. Our first iteration had multiple tabs of grid views, and a master/detail view. It worked, but had the obvious drawback of having to excessively scroll vertically to simply navigate around. After feedback (and using the app ourselves), we simplified the navigation dramatically and created multiple horizontal scrolling collection views on the main screen, and a “Search” screen for quickly filtering for known content.

Result: