University Affiliates by Location

Australia

01/12/24 - Met up with Tom for a casual chat about Pluralis. There seems to be some overlap with what he is doing and the compression front that Sameera is working on. Working on arranging a follow-up meeting with him for more details, as this first meeting was just a catch-up.

11/12/24 - Gil and Yan met up with Tom to chat in more detail about collaboration. Tom is working a lot these days on the above. He is looking at various applications related to fast approximate nearest neighbor graphs:

• FANNG for search and retrieval
• Search is now faster, >10M QPS, returning 32 near neighbours with 1-recall@1=0.9
• Graph building is now much faster. 100M items can be indexed in about 1hr40.
• He's been exploring k-means clustering and k=40M, N=1B gave 5 mins per iteration – so < 1hr for 10 iterations which seems to be enough to get plausible looking clusters – although I haven’t done search experiments yet to show that the clusters work for large scale retrieval.
• His postdoc Markus has integrated FANNG into pytorch so you can provide reference keys and index as well as search queries from pytorch.
• This technology has a lot of possibilities (e.g. access your training set in the inner loop at runtime instead of using it to train weights, curate the database to remove (e.g. repugnant language) or add (e.g. new knowledge – current events etc). Think RAG on steroids.
• Also looking at doing efficient training of models using hypersphere projection for better conditioning and using quadratic activations. Said he could train models on ImageNet in a single epoch and the conditioning of the models was much better using this approach. Might be relevant for the async pipeline parallel stuff AJ is looking at w.r.t stale gradients and corrections.
• Talked about inserting heads for certain stages in the pipeline to mitigate stale gradient issue but need to think more about how to do this in practice to prevent the model from generating degenerate gradients.

He’s keen to collaborate and open to the idea of working together on a shared research problems. Tom mentioned he would like to meet and talk with Alex, Yan or Gil will set up an introductory email to connect the two. |

United Kingdom

United States

California

Georgia

Massachusetts

New York

Pennsylvania

Indiana

Maryland