Not yet the new normal

Snapshot for Silicon Valley

Having been asked to do a status report on the Valley for compatriots in Scotland, this is a compilation of references for further reading.

More references, after completing the interview with Christine Esson for the Scottish Business Network insights series. Will update again when the recorded interview is available.

The California Consumer Privacy Act, discussed on the China Law blog.

A summary of a representative Venture Capital viewpoint from the point of view of the investors, rather than from founders looking for investment.

Elad Gil’s advice for founders.

Linley Spring Processor Conference Goes Virtual The Linley Group held their conference which gathers good quality marketing information about imminent AI hardware developments on Zoom, to excellent effect. They report 700 part-time attendees, compared to the usual 300 maximum in person for the event. Listening to it it seemed that the quality of technical information imparted was at least as good as it would have been at an in-person event.

Sherwood Partners doing two to five business windups per day. “This is the great unwinding,” Pichinson said. “We don’t know what’s happening, but we do know everything we believed in is changing. Everything we thought to be true may not be true.”

He expects the number of startup companies that he is ending from the coronavirus downturn to exceed the carnage left behind in the dot-com bust.

What I’m seeing isn’t layoffs – what I’m seeing is business continuing, though not business as usual. One company was acquired by a public company for cash, another couple of public companies are being acquired by much bigger companies, another needs advice about splitting off the original technology it started with, with the potential to re-capitalize and convert to a much more sustainable business model while compensating the original investors.

Delivery robots are out on the street in Mountain View

There have now been 100 reported deaths in Santa Clara County attributed to Covid-19 Vehicle traffic continues to be much lighter than before the lock-down; many more people are walking and cycling along the residential streets. We ride and walk for exercise, too.

Juiced E-bike, Trek road bike

Testing, tracing and tracking an outbreak in Seoul, South Korea. Paper It’s not clear when Santa Clara County is going to be able to implement this kind of tracing and follow up; without it removing the lockdown has a high likelihood of raising the case count and death rate again.

Call to action from Jim Yong Kim : social distancing, contact tracing, testing, isolation, and treatment are essential. Massachusetts is getting started on this.

ScaledML 2020 notes

26 – 27 Feb at the Computer History Museum in Mountain View

This was the 5th year of this conference. Metroid, who put it on, pick a useful mixture of academic and industry speakers, from people working at the front edge of getting products using machine learning into early adoption.  The most entertaining talk was from Josh Bloom, on applying ML in astrophysics. The most significant talks were from Jim Keller (Intel) on how Moore’s Law continues, and from Dennis Abts on his 14th chip design, Groq’s Tensor Streaming Processor. 

http://scaledml.org/2020/

——

Oren Etzioni Allen Institute for AI 

Paper https://www.technologyreview.com/s/615264/artificial-intelligence-destroy-civilization-canaries-robot-overlords-take-over-world-ai/ 

———

Metroid  turning research into product

Look for definition and understanding of ‘adequate accuracy’ for the context of the problem

Understand the rate of change of adequate accuracy – predictor of time to develop and investment requirements. Army of annotators available ? 

Can do model compression and optimization to match low capability edge AI chips. 

————-

Megan Kacholia VP engineering Google. https://ai.google/ 

https://www.blog.google/outreach-initiatives/accessibility/impaired-speech-recognition/

Tensor Flow TF 2.1 released Jan 2020 https://github.com/tensorflow/tensorflow/releases

————–

Andrej Karpathy Tesla https://www.tesla.com/autopilotAI 

Aiming for full self driving. Using the fleet (of customers’ cars) for data gathering.

——————

Ilya Sutskever Open AI 

Dactyl robot hand manipulation Rubik’s cube https://openai.com/blog/learning-dexterity/

Musenet for music . AI Dungeon game discussion at https://www.reddit.com/r/AIDungeon/

—————-

Wes McKinney Ursa Labs  forum for discussion https://discuss.ossdata.org/ 

Opensource support model Apache Arrow Looking for Swift and Julia developers

—————-

Savin Goyal Netflix Framework for AI development https://metaflow.org/ opensource

Sandbox, free, at AWS

———–

Panel discussion.  Another framework https://mlflow.org/  MLsys conference 

————–

Posters : Pure Storage;  Logical Clocks (Sweden) ; Samsung Iot chip, no system design

———–

Ion Stoica UC Berkeley  https://rise.cs.berkeley.edu/ 

————

David Aronchick Microsoft Leads open source machine learning at Azure

https://www.davidaronchick.com/  david_aronchick@microsoft.com

Structured schemas required for ML Ops. Design and test discipline for both data and algorithms.

———–

Joshua Bloom UC Berkeley 

https://bids.berkeley.edu/events/physics-machine-learning-workshop

Towards Physics-informed ML Inference In Astrophysics 

Searching for Planet 9   One hot encoding

Physics informed deep learning https://arxiv.org/abs/1711.10561

Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations:  Maziar Raissi, Paris Perdikaris, George Em Karniadakis

Reverse-Engineering Deep ReLU Networks https://arxiv.org/abs/1910.00744

David Rolnick, Konrad P. Kording

——————-

Matei Zaharia Databricks Scaling Machine Learning Development with MLflow 

More ml devtools . Reproducible runs Auto logging to support data versioning

———

Jim Keller Intel 

Moore’s law continues  1000 scalars. Abstraction layers are critically important 

Extreme ultraviolet lithography is the next phase of chip manufacture.

EUV is a step function enables 100x finer printing

Once you have stable data and a stable platform, the platform can evolve from CPU to GPU to special purpose accelerator.

————–

Josh Romero Nvidia Scaling Deep Learning on the Summit Supercomputer 

Used Horovid (Uber) framework for DL training.  Needed hierarchical all reduce 

——–

Peter Mattson Google MLPerf: driving innovation by measuring performance 

Need benchmarks for training, inference, mobile. Hard to get contributors. MLCommons non-profit formed to encourage innovation. People’s Speech dataset aiming for 100k hours of transcribed speech by diverse speakers.

Sean Lie Cerebras Wafer-Scale ML 

——–

Dennis Abts  Ditching the ‘C’ in CPU: Groq’s Tensor Streaming Processor (TM)

https://research.google/people/author36240/

Dataflow in a superlane 220 Mibytes shared SRAM

Memory has an address and a direction 1 Teraop/sec/mmsquared

Deterministic instruction time INT8, FP16 

14th chip.  No arbiter, no replay mechanism, no flow control in chip, no hardware interlocks – orchestrated by the compiler. 

Click to access Groq-Rocks-NNs-Linley-Group-MPR-2020Jan06.pdf

Slides and video are now available at https://info.matroid.com/scaledml-media-archive-preview Matroid ask for an email address in exchange for access.

———–

Understanding the new business of AI

Adding to the agreement and reaction to the useful Andreesen Horowitz post on The New Business of AI from last week ..

Image from the 2017 AI Index, Stanford Institute for Human -centered AI

Reaching the sunlit uplands of AI is going to be rather harder than many of its investors and protagonists have predicted. https://a16z.com/2020/02/16/the-new-business-of-ai-and-how-its-different-from-traditional-software/

In particular, many AI companies have:

  • Lower gross margins due to heavy cloud infrastructure usage and ongoing human support;
  • Scaling challenges due to the thorny problem of edge cases;
  • Weaker defensive moats due to the commoditization of AI models and challenges with data network effects.

Adding a couple of examples to illustrate some of the systems design issues :

Frank Denneman, from VMware, on paralellism used for scaling training models https://frankdenneman.nl/2020/02/19/multi-gpu-and-distributed-deep-learning/

Emily Potyraj, from Pure Storage, on optimizing ECG data layout to improve deep learning training performance. https://towardsdatascience.com/what-format-should-i-store-my-ecg-data-in-for-dl-training-bc808eb64981

All of this points to a continuing requirement for a high degree of skilled problem analysis and systems design in order to make best use of AI/ML . There’s an opportunity for existing services companies to dramatically improve with judicious use of ML/AI .

AI and new jobs

Last year when we were preparing for the AI and ML panel at the Markets Group meeting, we spent a lot of effort to prepare for questions on potential and actual adverse effects – but no-one asked. The audience were institutional investors, many of them managing pension funds for employees, so we really had expected pointed questions about the potential for removal of existing jobs and about how new occupations might arise.

Prompted by a blog post from Timothy Taylor, and quoting from a paper titled ‘The Wrong Kind of AI’ , it seems useful to think “about the future of work as a race between automation and new, labor-intensive tasks. Labor demand has not increased steadily over the last two centuries because of technologies that have made labor more productive in everything. Rather, many new technologies have sought to eliminate labor from tasks in which it previously specialized. All the same, labor has benefited from advances in technology, because other technologies have simultaneously enabled the introduction of new labor-intensive tasks. These new tasks have done more than just reinstate labor as a central input into the production process; they have also played a vital role in productivity growth.”

References

IZA DP No. 12292 Institue of Labor Economics The Wrong Kind of AI?
Artificial Intelligence and the Future of Labor Demand APRIL 2019
Daron Acemoglu MIT and IZA
Pascual Restrepo Boston University

More machine learning – ScaledML

27 – 28 March 2019

The ScaledML conference is growing up; from a Saturday at Stanford to a two day event at the Computer History Museum with sponsors. http://scaledml.org/2019/

Two big new themes emerged

  • Concern for power efficiency (Simon Knowles, Graphcore, talked about Megawatts; Pete Warden, Tensorflow talked about milliwatts and energy harvesting
  • Development platforms – Adam D’Angelo, Quora, was particularly clear on how Quora operate development to efficiently support a small number of good developers

David Paterson gave the first talk on Domain Specific architectures for Neural Networks – an updated version of this talk https://cacm.acm.org/magazines/2018/9/230571-a-domain-specific-architecture-for-deep-neural-networks/fulltext

The roofline performance model is a useful way to visualize comparative performance. For future performance improvements functionally specific architectures are the way forward; this requires both hardware updates (what Google is doing with the TPUs) and improved compiler front and back ends.

Fig 3 from the Domain Specific Architectures paper linked above.

Intel recognizes this trend – Wei Li described the work his team is doing to incorporate domain specific support into Xeon processors. This blog post has the gist of what he presented.

Most of the talks are here on YouTube

Machine Learning snapshot, June 2018

Kian Katanforoosh and Andrew Ng have been teaching CS230:Deep Learning, at Stanford. The project reports and posters list has just come out, summarizing work done by the students with help from the teaching team. More than 160 projects. It will be interesting to see which of these mature into applications.

outpainting

Image from Painting Outside the Box: Image Outpainting with GANs (Mark Sabini, Gili Rusak) which was awarded first place in Outstanding Posters

CS230 https://web.stanford.edu/class/cs230/

Reports and posters http://cs230.stanford.edu/proj-spring-2018.html

The 4th Research and Applied AI Summit  https://raais.co/ just finished in London. The 125 slideset on the State of AI is a decent current snapshot of much more evolved work than the Stanford posters. https://www.stateof.ai/

Artificial Intelligence for Institutional Investors

natpressclub

Venue – the National Press Club, 13th floor ballroom

The Markets Group runs events for institutional investors. I took part in a panel about Artificial Intelligence and Machine Learning, and in a round table discussion.

Diagrams and useful background sources listed below.

Definition of Artificial Intelligence, to illustrate that machine learning is a subset of AI. Sourced from the thorough review of AI in the NHS  http://www.reform.uk/wp-content/uploads/2018/01/AI-in-Healthcare-report_.pdf

AIdefinition

Examples of AI and ML in use

At least 75% of the audience uses Netflix – which applies machine learning to improve its users’ experience of streaming as well as for content selection. Its results are driven by an extreme emphasis on keeping existing and attracting new customers. The data its customers generate by using it are used to make recommendations to them. Artwork personalization https://medium.com/netflix-techblog/artwork-personalization-c589f074ad76  Image discovery https://medium.com/netflix-techblog/ava-the-art-and-science-of-image-discovery-at-netflix-a442f163af6

View at Medium.com

Alibaba and Tencent Unlike the US companies, they have built their support for retailing primarily based on the customer’s mobile device; so they use facial recognition for identification, image scanning and matching for item selection, precise location specification in shopping venues and integration with payment apps to enhance the customer’s buying experience, both online and in person at a store.  http://technode.com/2018/02/14/alibaba-new-retail/ http://www.bain.com/publications/articles/embracing-chinas-new-retail.aspx (joint report – Bain and Alibaba)

Recent reviews – longer reads  Recent developments are a direct result of the enormous improvement in computing capability at drastically reduced costs, a direct result of Moore’s Law – which continues.

Longer term effects of automation – 10 – 20 year horizon – Bain March 2018 http://www.bain.com/publications/articles/labor-2030-the-collision-of-demographics-automation-and-inequality.aspx

13 artificial intelligence trends reshaping industries and economies – CBInsights February 2018 https://www.cbinsights.com/reports/CB-Insights_State-of-Artificial-Intelligence-2018.pdf

Capabilities at the end of 2017 – summary in slide format from Jeff Dean (Stanford, Google Brain team) http://learningsys.org/nips17/assets/slides/dean-nips17.pdf

History

1950 Alan Turing asked “Can Machines Think”  Computing Machinery and Intelligence paper https://www.csee.umbc.edu/courses/471/papers/turing.pdf
1956 Initial definition of Artificial Intelligence at a workshop  at Dartmouth College https://en.wikipedia.org/wiki/Dartmouth_workshop   Remember the AI Winters in the 1970s, 1990s