Build Periscope in 10 Minutes

November 30th, 2015 by Chris Allen

Create_header_2

With live streaming becoming increasingly prevalent in 2015, developers are focused on creating applications to address the public’s fascination with streaming media. Periscope is the prime example of such an application and the sheer size of Periscope’s user base and class-leading engagement metrics validate its dominance in the space.

But what does it take to build a live streaming and communication platform such as Periscope, with the capability to broadcast to one hundred thousand or even one million subscribers? What if I told you that you could build a live streaming application with Periscope-like functionality and scalability in just 10 minutes?

Before we created Red5 Pro it took some serious effort to build this kind of server-side infrastructure and tackle the high level of complexity to build a native Android and iOS video encoder/decoder that works with the server. We saw this trend of a new kind of mobile app that connects people in real-time, and we saw these early adopters cobble together inefficient software wasting tons of time and energy. We couldn’t allow this to happen anymore, so we decided to make it easy for developers. With Red5 Pro, you truly have the ability to build the guts of the next live streaming phenomenon in a matter of minutes, and here’s how:

Let’s first start with all the pieces, and what you would need to build if you were to do this from scratch.

The Fundamentals

1. Publish from the mobile client:

  • Access the camera

  • Encode the video

  • Encode microphone data

  • Negotiate a connection with a media server

  • Implement a low-latency streaming protocol for transport

  • Stream the data to the server

2. Intercept with a media server

  • Intercept the stream

  • Relay to other clients

      and/or

  • Re-stream to a CDN (adds latency)

  • Record the stream (optional)

3. Implement client side subscribing:

  • HLS in WebView (even more latency)

and/or

  • Setup connection with media server

  • Implement streaming protocol

  • Mix the audio and video

  • Decode video/audio

  • Render video and play the audio

 

*Note-this is actually a simplified list of all the tasks involved. Try doing this on multiple threads and getting it to perform well; it is complicated! It’s truly a rabbit hole that most developers don’t want to venture down. Given the awesome tools and libraries that exist for us developers, we thought that it was ridiculous that an easy-to-use and extensible live streaming platform just didn’t exist. That’s why we built Red5 Pro.

 

Red5 Pro to the Rescue

Let’s uncomplicate this. The Red5 Pro Streaming SDKs provide what we think is an intuitive and flexible API to remove the complexity while retaining tremendous control if you need it. Let’s take a look at the classes our SDKs provide. (note that they are the same on Android and iOS).

Graph

Let’s step through an example using these classes, piece by piece.

The Publisher

R5Configuration:

Red5Pro_tools

The first step is to create an R5Configuration. This class holds the various data used by your streaming app. It contains things like the address of your server, the ports used, protocols, etc. In this example we are connecting to a server running at 192.168.0.1 on port 8554 via the RTSP protocol. This Red5 Pro server has an app called “live” running on it, and that is what we want to connect to based on the context name. And finally we are setting the buffer time to half a second.

iOS

 

Android

 

R5Connection:

Red5Pro_links

Next, you create an R5Connection object, passing in your configuration. This establishes a connection to the Red5 Pro media server.

 

iOS

 

Android

 

 

R5Stream:

Red5Pro_stream

Now you create a stream object passing in the connection object you just created. Note that the R5Stream is also used for incoming streams, which we will get to in a bit.

 

iOS

 

 

Android

 

R5Camera:

Red5Pro_camera

Next we create a camera object and attach it to the R5Stream as a video source.

 

iOS

 

 

Android

 

 

R5Microphone:

Red5Pro_microphone

Then we create a microphone object and attach it to the stream as an audio source.

 

iOS

 

 

Android

 

 

R5VideoView:

 

Red5Pro_view

While it’s not a requirement to publish a live stream, we find it useful to provide a preview for the user of their video being streamed from the camera. This is how you set that up.

 

iOS

 

 

Android

 

R5Stream.publish():

Red5Pro_publish

Finally the last step for the publisher is to publish the stream using a unique name that the subscriber can subscribe to.

 

iOS

 

 

Android

 

The record type parameter tells the server the recording mode to use on the server. In this example we are setting it to live, meaning it won’t record the stream on the server.

 

Here are your other choices.

R5RecordTypeLive – Stream but do not record

R5RecordTypeRecord – Stream and record the file name. Replace existing save.

R5RecordTypeAppend – Stream and append the recording to any existing save.

If you compiled and ran this app with it configured to point to a running Red5 Pro server, you would be able to see it running in your browser. Open a browser window and navigate to –> http://your_red5_pro_server_ip:5080//live/streams.jsp to see a list of active streams. Click on the flash version to subscribe to your stream.

 

LiveStreaming.png

The Subscriber

Now that we’ve built the publisher we have established a live stream being published to the server. Yes, we did see the stream in Flash, but in order to consume that stream on mobile we need to build the subscriber client. Let’s dig into that now.

 

R5Configuration:

Red5Pro_tools

Just as before, we setup a configuration object holding the details of our connection and protocols.

 

iOS

 

 

Android

 

 

R5Stream:

Red5Pro_stream

Then, like in the publisher, we set up a stream by passing in the configuration into the constructor.

 

iOS

 

 

Android

 

R5VideoView:

Red5Pro_view

This is the step where things deviate just a little from the publisher. We still set up an R5View, but this time we want to use it to display the incoming stream

 

iOS

 

 

Android

 

 

R5Stream.play():

Red5Pro_play

Finally, we tell the stream to play by using the play method and passing in the unique stream name that the publisher is using.

 

iOS

 

Android

 

Voila, you can now build your own one-to-many live streaming experience, all within minutes with the help of Red5 Pro. What do you think, are there ways we could make this even easier? We love hearing feedback, so let us know in the comments or email us directly. Happy Coding!

, , , , ,

Android – Live Wallpaper part 2

March 22nd, 2012 by Paul Gregoire

Let’s make the star that we created in part 1 rotate in just a few simple steps. Although before we start, I’d like to note that we are technically rotating the entire canvas and not just the star itself; adding more than one star to the display will clearly expose this minor detail. I’ve done some research on the rotation of multiple individual items at-once, but I have not yet found a solution that fits; feel free to comment if you are aware of how to accomplish it.

The code modifications below are being in the LiveWallpaper.java source file, we are not using the only other source file GoldstarActivity.java at this time.

1. Add a variable where we keep track of the first frame drawn. This will prevent some calculations from being executed more than once per instance.

2. To rotate the star, we will use degrees. In steps 2 and 3, two methods of generation are shown. In this section we use a float counter for degrees of rotation.

3. In this section we use a fixed array for degrees of rotation. To do so, we also must create an index counter and an array of 360 floats; again, its up-to-you as to which option to use.

4. If using the array method, the onCreate method is modified to pre-fill our array when the application is first initialized.

Note: I’ve tried both methods for degree cycling and they seem equally fast on my device.

4. Modify our drawing code from part 1. The code block is here for reference.

Before

After (using float counter)

After (using float array)

6. As an aside to what we done thus far, you can easily add another star with a couple extra lines like so:

The additional star will be down and to the right of the primary star. This star will not rotate per-se but instead will “orbit” the primary star.

7. Last step, build and run in the emulator. Right click on the goldstar project in Package Explorer and select Run As -> Android Application. If you haven’t created any AVD (virtual devices), you’ll be prompted to create one. Creation of an AVD is covered here.

When everything works, you’ll see this in the Eclipse console:

I recorded the emulator running the apk in this screencast

Running on my Galaxy Nexus

In the sample code, I’ve refactored the original class alittle to make things more clear.
Project Source

End of part two.

, , , ,

Android Graphics and Animation: Part II – Animation

July 19th, 2011 by Keith Peters

Android Animation

In the first part of this series, we covered basic graphics in Android – starting a new Android project, creating a custom view and displaying it, and using that view to draw custom graphics in its onDraw method. To recap, the drawing occured only when the onDraw method was called by the system when it determined that the app needed to refresh its display. This generally occurs once when the app starts and only occasionally, if ever, thereafter. For animation, we need to be able to trigger redraws on a regular basis. This is quite a bit more complex than drawing a static image, but not horribly so, so let’s dive in.

SurfaceView

In the last example, we extended View for our custom view class. That was fine for the purpose, but will not be adequate for drawing multiple times like we need to do for animation. For View, the onDraw method is triggered by the system when it knows that the Canvas is safe to draw on. It can set things up for us before calling onDraw, and then clean up when it is done executing. Since we need to do drawing on our own schedule, we need a view class that will let us do this set up and clean up ourselves. That class is called SurfaceView. So that’s what our new view will extend.

To get started, create a new Android project the same way we did last time. Call the project and activity “Animation”. Again, in the main activity class replace the call to setContentView with a custom view. We’ll call this AnimView:

Of course, AnimView does not exist, so we’ll get an error. Trigger a quick fix, which will offer to create the AnimView class. Before accepting the defaults in the New Java Class dialog, change the superclass field to “android.view.SurfaceView”. When the class is created, trigger another quick fix to create the constructor. You should end up with the following:

At this point, the app should compile and run, but naturally will show just a black screen.

SurfaceHolder

Again, since we will be a lot more in control of when things get drawn, we need to go a little more low level in what we are doing. When using onDraw, you are automatically passed a Canvas object that you are safe to draw on. When using SurfaceView though, you need to get your canvas from something called a SurfaceHolder. This can be retrieved by simply calling getHolder() from the SurfaceView instance. That’s easy enough, but there’s another bit of complexity coming up.

You can’t draw to a surface of a SurfaceView/SurfaceHolder until the surface is created. And you should not draw to it after it has been destroyed. So we need to know when these things happen. To do that, we can let the holder know that we want to handle related events. To do this, we call surfaceHolderInstance.addCallback(viewInstance). But one more catch – the object you pass to this method must implement an interface defined as SurfaceHolder.Callback. So our class definition starts out as:

public class AnimView extends SurfaceView implements SurfaceHolder.Callback {

When you do that, you’ll be informed that you are not implenting the required methods of that interface. Use a quick fix to add them. With all that done, you should have the following:

Threading

Now we can start animating. In animation, you need generally have some kind of model of what you are animating, with some kind of rules on how that model changes. you need to update the model, and then render that model to the display, then update the model again, render again, and so on.

If you are used to animating in Flash you’re familiar with doing this via enterFrame, or perhaps with timers. Timers are also used in JavaScript animation. In Android though, we generally use threads.

Threads can be a bit scary as they are a bit more complex than a simple timer. If you’re not familiar with threads, the concept is just that you are starting another process that runs independently from the main process. This is useful for operations that might take a long time or will not return immediately. The new thread does its own thing in its own time frame, and the main process of your app continues to do what it needs to do, remaining responsive, etc.

The scary part of threads is that they run separately, but are able to access the same variables and objects in a non-synchronized way. Thus, one thread might be performing some procedure on a given object, and right in the middle of tht procedure, the other thread might step in and change the state of that object or even delete it. So you have to take some extra steps to guard against these types of situations.

Our view will use a separate thread to perform its animation. We will create and start the thread running in the surfaceCreated method, and we will stop the thread in the surfaceDestroyed method. There are a number of different ways to use threads. The way we’ll do it is to subclass the Thread class and put the custom functionality in that class.

Here’s the start of our custom thread class:

In order to draw to a canvas, we’ll need the surface holder to get the canvas from, so we’ll pass that in in the constructor and save it. We’ll also need a variable that will indicate whether or not the thread is currently running and a way to set that.

When we create an instance of this thread class and call start() on it, its run method will be executed in a separate process. We’ll actually use a while loop to do our animation. This may seem odd if you’re coming from the Flash or JavaScript world, where in infinite while loop would just lock things up. But because this is in a separate thread, it works out fine.

The pseudocode for what we will do is like this:

This will just run forever. Well, until we set running to false anyway. As you might have guessed, we’ll create and start the thread in the surfaceCreated method and we’ll set running to false in the surfaceDestroyed method. There’s a few more details to it, but we’ll get there eventually.

Locking and Unlocking

To get a canvas from a surface holder, we actually call holder.lockCanvas(). This prevents anything from happening to the canvas while we are using it. When we are done with our drawing, we call holder.unlockCanvasAndPost(canvas), passing in the canvas instance we just drew to. This frees it up and displays what was just drawn.

Here is the final code with some actual animation going on:

Here you can see we declare the canvas variable, then we enter a try block where we get the canvas and do the drawing. This allows us to unlock the canvas in a finally block, so that even if an exception is thrown while drawing, we won’t leave the canvas in locked state.

Note that the drawing is done in a synchronized block. This puts a lock on the holder so that nothing else can change it from another thread while we are using it. In this block we set the background to black and draw a white circle. The x value will be incremented on each loop, moving the circle across the screen.

Starting and stopping the thread

All we have to do now is create, start, and stop this thread. We’ve already said that we’ll do that in surfaceCreated and surfaceDestroyed methods. So let’s see what this looks like. First the created:

Simple enough. We create the thread, passing in the suface holder, set running to true, and start it. This will wind up executing the run method, which will run that for loop in a separate process.

The destroyed method is a bit more complex:

First of all, we set running to false. This will allow the while loop in the run method to exit. But since that’s happening in another thread, we don’t know exactly when that’s going to happen. So we want to make sure that it’s really fully complete before we leave here. We do that with the join method of the thread. That will cause execution to stop and wait for that thread to end. However, this will sometimes result in an InterruptedException. So we throw that whole thing in a try/catch statement and keep retrying it until the join finally successfully returns. Here’s the final AnimView class:

, , , ,

Android Graphics and Animation: Part I

June 27th, 2011 by Keith Peters

This is the start of a series of tutorials on graphics and animation on the Android platform. There is plenty of information out there on how to create general form-based, controls-and-layout type of Android apps, but very little on how to do more creative drawing and animation. So this series will cover the following topics:

1. Android graphics.
2. Android animation.
3. Android input: Accelerometer.
4. Android input: Touch.

Today we’ll get started with simple graphics. There are actually a few different ways to draw graphics on the screen in Android.

First, there is the Canvas class, which gives you a nice basic drawing API to create lines, circles, rectangles, fills, strokes, deal with bitmaps, etc.

Then there’s OpenGL. If you’re going to do 3D or just need more raw graphics and animation power, you’ll probably want to use OpenGL, or more likely use one of the various 3rd party libraries that make it a bit easier to use.

And then there is something called RenderScript, which was introduced in Android 3.0 (which, at the time of this writing is supported by only a few devices).

For this set of articles, we’ll be using the simplest and most widely available option, Canvas.

Setting up an Android coding environment

Of course, before we can even get started, you’ll need to have an Android coding environment set up and a connected Android device. You could use the Android simulator, and you should use it for testing different device resolutions and capabilities, but in general day-to-day dev, you’ll probably find it faster and easier to deploy and test on a device.

I’m not going to go into very deep detail about this, only because Google has covered it in far more depth than I ever could. So I’ll just point you to the right place.

http://developer.android.com/index.html

Here you’ll find links to the SDK, Developer’s Guide, References, Resources, Videos, and a blog. Within all that, you’ll find step by step instructions on how to set up your environment. But in a nutshell, you’ll need to:
1. Install Eclipse (or another editor of your choice, but this tutorial will assume you’re using Eclipse).
2. Download the Android SDK. This is just a folder of files and tools used in developing Android apps.
3. Install the ADT Plugin, Android Development Tools. This is an Eclipse plugin that will set up your Eclipse install to build Android apps.
4. Add Android platforms and components.

These steps are all covered in more detail here:

http://developer.android.com/sdk/installing.html

Connecting a device or creating a virtual device (emulator)

Next you’ll need to have someplace to run your code. Again, I recommend using a real device as much as you can. Setting up a device for development is covered here:

http://developer.android.com/guide/developing/device.html

If you don’t have a physical device, or are at a point where you need to test some different resolutions or features your device doesn’t have, this link will walk you through setting up a virtual device on the emulator:

http://developer.android.com/guide/developing/devices/index.html

Code!

OK, let’s make an app. Assuming you have everything installed and working, and are using Eclipse as your editor, fire it up and create a new workspace. Then create a new project by using the menu File -> New -> Android Project. This will bring up the “New Android Project” dialog.

Give your project a name, “Drawing” and choose a Build Target. We’ll stick with Android 2.2 since that’s a pretty common one.

Going further down, we need an application name, package name, and activity name. The application name is what will show up on the device. For now, think of the activity name as the name of the main class of the app. The package is the class package as in any Java project. Finally we need to specify the minimum SDK version. We’ll choose 8 here to coincide with the Android 2.2 SDK. The whole numbering system for SDKs and SDK versions is a bit confusing. I’ll leave it to you to figure it out more on your own. But the above settings will work for now.

Now we can click “Finish” and our project will be created. Your package explorer view should look like this:

There you can see your src folder with your package and main activity class. Opening that class you should see the following code:

Since this is the only activity in this application, this class will be instantiated when the app is run, and the onCreate method will be called. This is where you want to hook into to initialize pretty much everything.

Right now, onCreate calls super.onCreate and then setContentView, passing in something called R.layout.main. If you’re curious what this is, look in the folder res/layout and you’ll see main.xml, which will look like this:

If you’ve done any work with Flex, Silverlight, or any other XML-based layout systems (or even HTML) this will look pretty familiar. It creates a layout with a single child that is a TextView. The TextView’s text property is set to “@string/hello”. If you want to see what that is, look in res/values/strings.xml.

The Android compiler will compile all the stuff in the res folder into classes or embeddable assets as appropriate. So res/layout/main.xml becomes the R.layout.main, which is an instance of a class that extends View and can be set as the activity’s content view using setContentView.

Now, if you’ve set everything up correctly, you should be able to run or debug this project on your device and/or in the emulator and see something like the following:

If this is not working, stop here and get it debugged. This is the bare bones of project setup, and everything else depends on this.

Custom Views

OK, that’s all very interesting, but we’re not going to use much in the res folder or any of that xml-based layout stuff here. We’re going down to the metal and writing our own drawing code.

But since we aren’t relying on the compiler to create a view from xml for us, we’ll have to make our own view class. We can even use some of the ADT plugin’s shortcuts to let it do a bunch of the work for us. Change Drawing.java to look like this:

Here we’ve created a new class member, drawingView of type View, instantiated it as a new DrawingView, passing in this to the constructor, and set it as the content view.

Of course, Eclipse will complain because DrawingView does not exist yet. But if we click on that error it will offer to create the class for you. It will even know that it should extend View. So go ahead and let it create that class. It should look like this:

Now it’s going to complain again because it wants a constructor that takes an argument. Again, use the quick fix feature to let it create the constructor it wants. Now you’ll have this:

We’re at a stable point here, so go ahead and run that on your device/emulator and make sure it launches. You shouldn’t see anything but a black screen with the app name at the top, but it should compile and deploy.

OK, now we have a view we can draw in. The View class is designed so that all the drawing will be done in an onDraw method. This method will be automatically called whenever the view needs to be redrawn. To create this method, type “onDraw”, trigger auto-complete, and accept the first choice. You should wind up with an onDraw method like you see below (or you could go all old school and actually type it by hand).

You see this method has given us a Canvas to draw on. If you trigger autocomplete on canvas, you’ll see that it has all kinds of drawing methods. Let’s add a call to drawLine right after the super.onDraw call:

As you probably guessed, the first arguments for this are the x, y values of an initial and an ending 2d point. The last argument, paint, is a Paint object that tells the system what to make this line look like (color, width, etc.). Since we haven’t defined paint yet, it will give you an error. Trigger a quick fix to create a field named paint. Then in the constructor we’ll instantiate it and give it some properties. Here’s the result:

Don’t forget the imports for Color and Style. You can run or debug this now and you should have an utterly fascinating diagonal white line on your device’s screen. When you’ve calmed down and gotten yourself under control, we’ll move on.

Setting the Background Color

Perhaps you want to change the background color. You can do that will canvas.drawColor, passing in the color you want to use. Note that this will actually clear the screen, so you’ll want to do this before drawing anything important.

Specifying Colors

In addition to the constants on the color class, like Color.BLACK, Color.WHITE, Color.RED, etc. you can specify exact colors with Color.rgb(red, green, blue) where each parameter is an int from 0 to 255, or Color.argb(alpha, red, green, blue) if you need transparency.

So to set the background to a kind of light purple, do something like this:

Other Shapes

As mentioned, there are lots of other options on Canvas for drawing various things. A few examples:

Here cx and cy are the center point to draw a circle with the given radius.

Here rect is a Rect object or a RectF object (which would use floats rather than ints for its measurements).

Pretty obvious.

Then there are drawOval, drawArc, drawRoundRect, and many others.

Putting it all together

Just to implement a few things all at once, we’ll do something like this for a final demo:

Here we’ve set the style to FILL instead of STROKE, then use some fancy math and a couple of for loops to draw a grid of squares, each with a random color. Nothing amazing, but assuming you have some previous experience with any kind of drawing API from any other language, this should set you up to create all kinds of custom graphics in your Android app or game.

Summary

Here we’ve seen how to set up a new Android project and create a custom view that we can draw into. The view class is instantiated and added as the activity’s main content view, and the onDraw method is called when it’s ready to display.

Of course, since generally speaking this is only called the one time near the start of the app, it’s just a static drawing. In the next installment of this series, we’ll dive into animation and making things move in Android.

, , ,