CV and Sports

Collection of computer vision and sports research

[btrcvblog-blog]

What is CV

CV is a specific area of study within the boarder field of Artificial Intelligence. Just like how NLP (natural language processing is often regarded as it’s own field, separate from AI). CV enables computers to receive data, often in the form of images, but not exuding video (continuous stream of images), point clouds, x-rays, thermal and infrared scans. 90% of the worlds media is image based in one form or another and this is why CV is estimated to be a 48 billion dollar industry by 2022. 

Recently the field of AI and all it’s sub categories have been able to evolve due to two key advancements. One is pure computational power, the GPU in our laptops are more capable than the first super computer. the second is the amount of data available, especially clean and labelled data, it enables the computer to learn through it’s algorithms. CV can be simplified into one major task, pattern recognition. The computer needs to have a vast source of prior experience or data to draw on to perform the analysis. So given 100,000 different cat pictures, the computer now can understand and match any given cat picture to something it has seen before. 

“What Is Computer Vision.” Image Recognition: Image Recognition And How It's Used Today | GumGum, gumgum.com/what-is-computer-vision. 

[btrcvblog-blog]

GumGum

GumGum is a company that uses applied computer vision to solve various advertising problems in various domains. They’ve recently tackled to the sports industry and have received overwhelming results. The focus of analyzing video footage to advise investors on ad placement and efficacy. As you can see through the illustration they focus on a number of different factors including clarity, prominence, visibility, size, share of voice and placement. Ad placement of a logo verses the full company name is a difference between $334K and $495K in media value at the 2017 US Open. It is no surprise to anyone that they found social media influences trumping broadcasting ad dollars (81% of sponsored media value is driven by social media, while broadcasting is only 19%). The company also uses computer vision to scan social media and advise clients on their marketing campaigns. Although this is not using pose estimation specifically, this is definitely a prominent and lucrative use of computer vision in the industry of sports. 

“Measuring The True Value Of Sports Sponsorships with Computer Vision Technology.” SportTechie, 4 Apr. 2018, www.sporttechie.com/high-sponsorship-value-of-unofficial-social-media-accounts-gumgum-sports/.

“Pro Sports Teams Turn to Computer Vision to Vet Their Sponsors.” Data Makes Possible, 7 May 2018, www.datamakespossible.com/sports-teams-computer-vision-sponsors/.

“Sports Sponsorship Valuation | GumGum.” Image Recognition: Image Recognition And How It's Used Today | GumGum, gumgum.com/sports.

[btrcvblog-blog]

Tracking Players in Sports

CV in sports and specifically tracking players is a difficult problem and is a field that is still ongoing. It is defined as “automated capturing, analyzing and quantifying the ability of the athlete” in this suite of papers from the Machine Vision Laboratory, University of Ljubljana. 

There are some of the challenges presented in the field of sports:

rapid, often unpredictable movement

collisions and occlusions 

complex non-rigid forms

tracking of extremities 

large potential coverage area 

These issues are often tackled separately because the field is rather new and still developing. Most studies try and focus on one specific aspect and as of now there is no overarching comprehensive system that could account for all of these challenges. General classification can be for ball-sports, individual sports, team sports, in-door or out-door. 

*This is a broad overview of a suite of papers that will be further discussed in the annotated bibliography. 

Rogelj, Peter. Machine Vision Group - Tracking Players in Sports Games, vision.fe.uni-lj.si/RESEARCH/trackp/index.html.

[btrcvblog-blog]

CV/Sport Applications

Sports focused areas of computer vision include:

improving broadcast / viewer experience

improving training process

automatic sports analysis and interpretation

helping / improving referee decisions 

commercial benefits 

There can be overlays that are applied to help viewers understand the game better, as well as move the camera to help focus in on the action or game play. 

Using pose estimation, the center of gravity can be calculated. This can be used for adjusting form, coaching players and understanding mechanics of the human body. 

Tracking player location through SLAM (simultaneous location and mapping) can help understand tactics in specific sports and court usage. 

Virtual replay and line judging systems such as Hawkeye and VAR can help make final decisions or simply assist in making game changing decisions. 

Ads can be optimally placed for viewer impact and value. 

Key challenges presented are: changing environments, real-time performance, innovation and value, robustness and accuracy and public push back. 

“Computer Vision SPORTS. Presented by Alex Golts.” PDF, DocPlayer, docplayer.net/45549796-Computer-vision-sports-presented-by-alex-golts.html. 

[btrcvblog-blog]

Game Changer

STATS LLC is a sports analytics company that is works with some of the biggest and longest running computer vision driven companies. They collect the most data on sports and in a large number of different sports. Inspired by the true story of Money Ball, the company took off and viewed sports from the lens of statistics. The work with teams and individuals on different levels. For teams, they are able to use the vast of mount of data they gathered to advises them on game play and help understand opponents. Their approach is mostly data-driven so it heavily depends on the amount and the quality of data they gather. With that they are also able to predict outcomes and which factors contributed to those outcomes. They currently have the potential to greatly impact the sports gambling industry. 

“How AI-Based Sports Analytics Is Changing the Game.” ADTmag, adtmag.com/blogs/dev-watch/2017/07/sports-analytics.aspx?m=1.

[btrcvblog-blog]

Improving Posture with Depth Camera

Using Microsoft Kinect which has a build in depth camera, researchers from the university of Hong Kong has developed a more accurate way of detecting posture. They use classify behavior, kinematics, colour and depth as reliability terms to produce more accurate detection. The application has potential uses in smart offices, laborious work environments and hospitals.

Researchers from the University of Hong Kong are developing a framework for human activity monitoring that uses depth cameras (Microsoft Kinect) to determine 3D posture. New concepts that are introduced in this paper are a set of reliability measurements use to produce increasingly more accurate joint detection. The reliability terms used include behavior, kinematics, colour and depth. 2D analysis techniques used in the past lacks sufficient information and motion capture techniques required extensive set up. This new framework produces more reliable results through machine learning algorithms that combining depth information, context of the activity, and available background knowledge of possible range of motion. The results are surprisingly accurate and can identify differences in posture that are imperceivable to the untrained human eye in seconds. Real-world applications of this technique could increase postural health in smart offices, reduce repetitive stress injuries for the workforce and aid in early detection of several musculoskeletal diseases. This framework can be applied to detecting mistakes in athletic form and provide valuable analysis of how injuries can occur. A cross discipline approach with the field of kinesiology and sports medicine could yield greater understanding of the human body.

Ho, Edmond S.L., et al. “Improving Posture Classification Accuracy for Depth Sensor-Based Human Activity Monitoring in Smart Environments.” Computer Vision and Image Understanding, vol. 148, July 2016, pp. 97–110,  doi:10.1016/j.cviu.2015.12.011.

[btrcvblog-blog]

CV and AssistiveTechnologies

Task-oriented approach towards assistive technologies(AT) allow a more direct bridging of core technical concepts than user-needs based approaches. Researchers are then communicate more fluidly and borrow useful concepts of from seemingly non-related projects. Applications include: wheelchair control through gestures, descriptive captioning for visually impaired users and fall detection alert systems. 

This article provides a brief overview of the how computer vision can be utilized in assistive technologies (AT). Traditionally AT is categorized based on “user-needs” but this article suggests structuring the research this way does not facilitate “cross-contextual applicability of most of the underlying techniques and methodologies involved in each specific context”. Instead, the articles provide a review of current research categorized using a “task-oriented” approach, comparing the technical tasks instead of the user needs. Mental functions, personal mobility, sensory functions and daily living activities are classes of user’s needs that are directly impacted by computer vision, as defined by the Global Cooperation on Assistive Health Technology (GATE). Applications that may require human activity recognition such as using gestures to control a wheelchair instead of physical forces or creating accurate descriptive captioning for those who are visually impaired are just a few examples of how computer vision advancements could impact AT. The task-oriented approach in this article showcases comparable computer vision techniques that can be translated from the field of sports to AT and vice versa.

Leo, M., et al. “Computer Vision for Assistive Technologies.” Computer Vision and Image Understanding, vol. 154, 2017, pp. 1–15., doi:10.1016/j.cviu.2016.09.001.

https://www.researchgate.net/publication/257694485_Fall_detection_system_using_Kinect's_infrared_sensor?_sg=oAyDJwwp6m93akI5x7wezIFoFlvZVcHCnDczxhKwuVsL8CHS-4Qvd_inJ3UZD9_Br4Vg6_i1UsC5JcTDsFAliJLVqK0MxO41NQ

[btrcvblog-blog]

Future Applications

Applications in other domains can benefit from research done in computer vision and sports. If programs could accurately estimate the human body and its motion, it could be used in gait analysis, fall detection, assertive technology for people with all kinds of disabilities and of course rehabilitation. The major advantage of using computer vision is that it is digital and therefore remote use is possible. 

Section 6.3.2 of Computer Vision for Assistive Healthcare focuses on posture and movement applications of computer vision and provides a summary of current known applications. Specific examples of applications include gait analysis, fall detection and movement for rehabilitation. Concepts and techniques used in related areas of research can be used to inspire new applications in the sports industry. Gait is linked to cognitive function and could be used as an early indicator of neurological issues. Potential application in a sports setting would be to detect concussions and other serious head trauma. Fall prevention requires a deeper analysis and predictive capabilities from the machine. Current research shows a combination of both computer vision and other sensory data such as sound can be successful in detecting falls. This technique can be applied to athlete’s movements by adding footsteps or other forms of sensory data such as force of impact, thermal mapping of body heat or weight distribution. The major advantage of using computer vision for rehabilitation is making off-site or remote monitoring possible, therapists and patients can interact digitally through cameras or even VR.

Marco, Leo & Farinella, G. Maria. Computer Vision for Assistive Healthcare. Elsevier Academic Press. Apr, 2018.

Tapu, Ruxandra et al. “A computer vision system that ensure the autonomous navigation of blind people.” 2013 E-Health and Bioengineering Conference (EHB) (2013): 1-4.

[btrcvblog-blog]

Physiotherapy Bot

Using open source technology OpenPose, a hack-a-thon group has developed a ROM assessment tool. Range of Motion is a key indicator for physiotherapists when they assess a patients muscle-skeletal health. Applications such as this one is now available for the public to create, with minimal funding and low hardware requirements. 

The 2017 Sydney HealthHack challenge was to use computer vision to determine range of motion (ROM), a measure often used by physiotherapists. Often an individual’s ROM is affected by physical injuries and can be used to gauge recovery. Computer vision can provide an automated way to assess ROM in real-time and help motivate patients with progress updates. The winning team of the HealthHack comprised of seven individuals stemming from various backgrounds, such as computer science, physics, and healthcare. An extremely successful prototype was created in one weekend using open source technology and without much funding. The current state of technological advancement in this area is high enough that these prototypes are readily available, they are ideas that need to be further refined. Athletes need regularly scheduled check-ups and undergo constant monitoring of their physical conditions, having a portion of those check-ups automated increases efficiency. ROM is just one aspect of the assessment process, there are many other factors such as speed of movement or duration or correctness of poses that could easily be implemented through computer vision.

Coviu. “Artificial Intelligence for Physiotherapy – Coviu – Medium.” Medium, Augmenting Humanity, 6 Nov. 2017, www.medium.com/@coviu/artificial-intelligence-for-physiotherapy-1f22fb4ac5f.

[btrcvblog-blog]

Wrnch

Wrnch one of the world’s leading teams pursuing understanding human motion through computer vision. They have developed a working platform for using only RGB video instead of using a RGB-D or RGB-Ir camera. They are not directly related linked to the sports industry, instead they focus on security, entertainment, health and wellness and automotive.

Paul Kruszewski, a computer scientist and CEO of Wrnch, joins Tech Emergence to explain his research on using computer vision and 3D modeling to have computers “understand” human movement. Dr. Kruszewski believes the main motivation behind computer vision related to body language is the need for increased safety and security. Some examples of potential applications include fall prevention for the elderly, detection of criminal activity, and pedestrian safety related to self-driving vehicles. Wrnch has produced an AI platform able to detect the human body from a simple video and is working on several related areas such as AR, VR and Motion Capture. There are numerous benefits and countless applications of computers being able to detect, label and analyze the human movement. The platform also has potential in the sports industry and could lead to huge advances in athletic performance, injury prevention and recovery when applied with sports medicine. All the areas of research mentioned above have different uses of computer vision, but the core technology used is very similar. The competitive nature of sports could also help drive funding towards this type of research.

(1)   Faggella, Daniel. “Computer Vision for Body Language - How It Works and How It Could Be Used -.” TechEmergence, 3 Dec. 2017, www.techemergence.com/computer-vision-for-body-language/.

(2)   “SXSW Accelerator Preview: WRNCH Is The Eyes For The Future Of AI.” Nibletz, www.nibletz.com/events/sxsw/wrnch-ai.

[btrcvblog-blog]

Physimax

Physimax is a company that uses computer vision to assess, rehabilitate and train current professional athletes. They use propitiatory technology and the Microsoft Kinect to achieve stunning results. The process is outlined as  “test, analyze, review results and customize exercise”. They currently work with the Indiana Pacers and has helped them become the number one team in the NBA to reduce injury related costs. 

Sanvada, a technology related content provider, highlights the benefits of using computer vision to keep athletes fit and ready for competition. The article touches on the competitiveness of the industry and the far-reaching consequences of an unprepared athlete. The article features an interview with the CEO of Physimax, Ram Shalev, the company uses computer vision to “boost athletic performance, prevent the next non-contact orthopedic injury and ensure quick and safe return to play”. Using specialized technology is still an emerging trend among professional athletes but it is steadily increasing. The main hardware used by Physimax is the Microsoft Kinect, the camera gathers the data used to generate movement profiles. Their process is broken down into four steps: “test, analyze, review results and customize exercise”. Physimax targets ball sports and offers fast turn-around times for each athlete. Early adopters to use this technology have been sports teams at the professional and collegiate level, as well as, the military. Physimax demonstrates the practicality of using computer vision to improve performance and reduce injury. This application has the potential to disrupt the sports industry as it gains popularity.

Hopton, John. “What Can Technology Do for Sports Stars?” Sanvada, Sanvada, 7 Sept. 2017, www.sanvada.com/2017/09/07/what-can-technology-do-for-sports-stars/.

[btrcvblog-blog]

AI in Sports

Computer vision is a specific field of study in the board field of artificial intelligence and this is a review of some applications of AI in the sports industry. From a business perspective the applications of AI and computer vision will improve spectator experience and ultimately increase profits. Additionally performance can also improve with the help of this new technology. Wearable tech is mostly sensor based and not categorized as computer vision but there are also innovations in that area in the sports industry. However, the push for computer vision innovation is because CV is less intrusive than attaching sensors in most situations. 

The future of AI in sports is a broad topic that spans several disciplines, from sports broadcasting to wearable technologies. This article from Tech Emergence provides a summary of current and potential future applications of AI in sports. Although not directly related to the use of computer vision in assessing body language, solid background knowledge about this field will help readers understand where different applications of computer vision belong in the rapidly changing industry of sports. From a business perspective the sports industry is projected to focus more on “enhancing spectator experience”. The ultimate business goal is revenue generation, but a beneficial side effect is the amount of funding for technological advancements. Current applications of AI include chatbots, computer vision, automated journalism and wearable tech. Future applications include AI Assistant Coaches, Smart Ticketing, Automated Video Highlights and Computer Vision Referees. In addition to improving the spectator experience, advancements in technology also help boost performance. U.S. Olympic Committee director, Mounir Zok stated “wearable tech contributed to Team USA earning 121 medals at the 2016 summer Olympic Games.” However, it is worth noting that these uses are not common due to the high costs associated. The results show that athletes and their teams are willing to support new innovative technologies to gain a competitive edge.

Sennaar, Kumba. “Artificial Intelligence in Sports - Current and Future Applications.” TechEmergence, 3 Mar. 2018, www.techemergence.com/artificial-intelligence-in-sports/.

https://youtu.be/KJlsvRGpQpU - PIQ

https://youtu.be/CUQG45juSN8 - Tennis In/Out

[btrcvblog-blog]

Overview: CV in Sports

Categories:  

            1) Camera calibration and tracking

            2) Detection and tracking

            3) Broadcast enhancement

Pose estimation is prevalent in detection and tracking but can also be used in other categories.  “Systems are currently limited to providing visualization of the player’s movement rather than detailed analysis, as accurate marker-less pose estimation is very challenging.” Some challenges that these applications have to over come is the accuracy, speed of computation, as well as speed of the physical movement, correct and appropriate model generation and occlusion. 

There are many such applications already in use, for example: VAR for Fifa, Hawkeye for badminton, cricket and tennis.

There are many data sets that are available specific individual sports, as well as, a variety of sports mixed together. Such as: Olympic Sports Dataset (16 sports), Sports Videos in the Wild (30 Sports) and UIUC2 (Badminton), and Soccer Video and Player Position (Soccer). 

This article published in Computer Vision and Image Understanding, highlights current applications of computer vision techniques in relation to sports and potential research areas for the future. Techniques are discussed based on their board categories: camera calibration and tracking, detection and tracking, and broadcast enhancement. Pose estimation is the specific focus that is heavily used, but not exclusively, in detection and tracking applications. The goal is to accurately detect and thus be able to create a model that represents human motion and utilize those models for further analysis. This is an ongoing area of research, as “systems are currently limited to providing visualisation of the player’s movement rather than detailed analysis, as accurate markerless pose estimation is very challenging”. Currently available commercial systems are listed and organized based on their primary focus in a table format that allows direct comparison of the methods, and datasets being used. It is well organized and provide solid starting points for further research and exploration.

Thomas, Graham, et al. “Computer Vision for Sports: Current Applications and Research Topics.” Computer Vision and Image Understanding, vol. 159, 2017, pp. 3–18., doi:10.1016/j.cviu.2017.04.011.

[btrcvblog-blog]

Leading computer vision scientist Fei Fei Li gives a board introduction of computer vision research. Keep in mind this is from 2015 and therefore is slightly out of date! However, it is a good overview of the computer vision problem. Computer vision identifying a set amount of object is different from visual intelligence, in order for a computer to understand scenes from pictures there needs to be added context. The example Fei Fei used was tell stories from pictures which requires natural language processing. When applied to sports, the algorithms need to first identify the human body and then find a context that is applicable.  

[btrcvblog-blog]

Research Questions

1) What different kinds of computer vision techniques are being applied to analyze the human body to try and accurately detect movement?

        *  How can these be techniques be applied in specific areas such as                       health, injury and performance?  

2) What are the benefits and drawbacks of using computer vision as a form of analysis for athletic movement compared to traditional assessment?

      * Should these new assessment techniques be combined with the                          traditional ones or replace them?

      * How effective are these techniques in specific areas such as skills                        acquisition, training, and rehabilitation?

3) How will advancements in the field of detecting accurate human movement impact the professional sports?

      * What other fields or applications could it also affect?

[btrcvblog-blog]

Understanding vision and building visual systems is really understanding intelligence...and by see, I mean to understand, not just to record pixels

Fei Fei Li