Date   
The majoy version of FATE is upgraded! Seven highlights of FATE v1.1

FedAI Admin
 

Hi Everyone,

 

Thanks for all your support. Our project FATE has reached the milestone of 1000 GitHub stars.

 

Besides, a lot of contributors from Tencent and other well-known enterprises, some from HKU and other universities, join into our open-source community. FATE will be better because of you.

 

Recently, version 1.1 of FATE is released officially. This new version provides a general algorithm framework and supports multiple federated learning algorithms. In addition, cooperated with Vmware, KubeFATE which is in partnership with Vmware provides fully containerized cloud native deployment. Now, the project has been launched on GitHub: (https://github.com/FederatedAI/FATE).

 

Introducing the several highlights of the updated version:

 

1.      FATE v1.1 provides a general algorithm framework supporting secure aggregation for homogeneous federated learning. We encapsulate the main process of homogeneous federated learning. The developers can be easy to implement their own homogeneous federated learning algorithm. For example, we use the framework to implement the homogeneous neural network algorithm.

2.      In this version, we add the Homogeneous Deep Neural Network, Heterogeneous Linear Regression and Heterogeneous Poisson Regression. It is very useful to use Heterogeneous Linear Regression in the scenario of predicting a continuous label. In addition, building models and forecasting in Heterogeneous Poisson Regression can help developers better to forecast the counts and frequency, such as the frequency of purchasing Insurance and the prediction of the accidents’ frequency.

3.      As a significant version, FATE has multiple algorithms and supports multi-host heterogeneous federated modeling, covering the binary classification, multiple classification, Regression. It can enable multiple data providers to train the federated model under a heterogeneous scenario.

4.      You can’t miss the FATE v1.1 if the developers already have the Spark clusters and they want to reuse preemptive resources. In this version, you can choose Spark as a computing engine to configure flexibly according to your actual situation. This is also an attempt to connect the Spark ecosystem.

5.      In the scenario of the online Inference, FATE now adds the heterogeneous SecureBoost to forecast. Meanwhile, FATE-Serving also adds service governance function. In other words, it will deploy multiple identical servers online. If you use the service governance function, the process will switch between the remaining servers when one server has problems. This function can prevent the data lost and other situations due to the failure of one server.

6.      At last, we are going to introduce KubeFATE which is in partnership with Vmware. In this version, KubeFATE provides fully containerized cloud native deployment for the developers. We encapsulate all components of FATE in containers and use Docker Compose or Kubernetes (Helm Charts) to deploy. The developers can easily deploy and use the FATE project by KubeFATE in the public cloud and private cloud.

 

Of course, we are much more than this. Let’s check out the poster to know more about the new functions of FATE v1.1.


Re: Query the server-client setting-up

Tianjian Chen
 

Hi Hank

FATE is easy to build and install (relatively) regardless whether you are doing a datacenter-datacenter vertical split federated learning or you are doing a client-server horizonta-splitl federated learning.

Please tell us your specific issue with build and deployment. 

//bests
Toby


On Mon, Oct 21, 2019 at 10:11 PM <hank9cao@...> wrote:
Hi all,
we are new here, would like to use FATE as the pooled analysis tool in our collaborative research (8 labs in Germany) in genetic psychiatry. We would like to know if the server-client setting-up of FATEis easy to build? We explored the TFF and PySyft packages which, unfortunataly, only supported the simulation of geo-distributed datasets, but we need to applied the pakcgae on the real machines, thus any support or tools for this are important to us.


Regards,
Hank

Query the server-client setting-up

hank9cao@...
 

Hi all,
we are new here, would like to use FATE as the pooled analysis tool in our collaborative research (8 labs in Germany) in genetic psychiatry. We would like to know if the server-client setting-up of FATEis easy to build? We explored the TFF and PySyft packages which, unfortunataly, only supported the simulation of geo-distributed datasets, but we need to applied the pakcgae on the real machines, thus any support or tools for this are important to us.


Regards,
Hank

The latest news of FedAI are updated! Let’s find out what happened in last month。

FedAI Admin
 

Hi All,

 

The latest news of FedAI has been released. Please find out what we happened in September.

 

 

1.        WeBank joins Linux Foundation, federated learning helps the innovative banking

 

On September 17th, the biggest international non-profit open-source organization Linux Foundation announced that WeBank has become its latest gold member. Currently, Linux Foundation members include Alibaba, Dell, Facebook, Toyota, Uber and other companies. WeBank is the first financial institution that acquires the gold membership.

 

It is reported that recently WeBank’s AI team led the transfer of the FATE (Federated AI Technology Enabler) to the Linux Foundation. FATE is the world's first industrial-level open-source framework, which is independently researched and developed by WeBank. As a gold member, WeBank’s AI team and global developers jointly promote the establishment of a federated learning ecosystem and develop and promote AI technology and its application under the protection of data security and user privacy.

 

(Please click here to find more)

 

2.        WeBank and Extreme Vision create the first vision federated learning system to promote federated learning in the field of visualization

 

On September 17th, WeBank and Extreme Vision officially signed a memorandum of understanding (MoU) in Shenzhen. The first vision federated learning system among the industries, jointly created by the parties, has been officially launched. As the first application of federated learning in the field of computer vision, the cooperation will focus on breaking through the three difficulties, data silos and missing data, data security, policies and regulations, which are caused by the competition of computer vision industry.

 

(Please click here to find more)

 

3.        WeBank and Tencent upgrade the cooperation, federated learning and Shield Sandbox jointly build the industry benchmark

 

Recently, Tencent Cloud – WeBank fintech innovation laboratory has officially established a federated learning joint research project of WeBank and Tencent Cloud Shield Sandbox. Both parties will conduct a series of product research and development and iterative optimization based on federated learning. Meanwhile, they will have deep cooperation in research and development and industry standards and so on, to promote the application implementation of AI technology under privacy protection.

 

(Please click here to find more)

 

 

4.        Tencent Data Security experts talk about the federated learning open-source project FATE: A bridge to the ideal future of privacy protection

 

Recently, FATE open-source community of WeBank has two new contributors – Yang Liu and Shuqi Qin from Tencent. As experts in the field of cloud computing security, they have constructed a new function point for FATE and submitted the related bug and fix it on the GitHub. It is reported that the core computing module of Tencent Shield Sandbox is provided by FATE. The sandbox project team will actively seek for methods to improve the shortcomings when they use the FATE framework and algorithm and make contributions to FATE open-source project. This form of cooperation also promotes the polishing of Shield Sandbox products and the improvement of FATE project.

 

(Please click here to find more)

 

5.        From IJCAI 2019 to NeurIPS 2019, federated learning will show up again in the international AI summit

 

At the middle of December in 2019, NeurIPS 2019 (Thirty-third Conference on Neural Information Processing Systems) will be held in Canada. NeurIPS is an International top academic conference on artificial intelligence, listed as A-class conference in the field of artificial intelligence by China Computer Federation. At this top conference in machine learning, WeBank, Google, Nanyang Technological University (NTU), Carnegie Mellon University (CMU) and other institutions jointly hold the Workshop on Federated Learning for Data Privacy and Confidentiality. More than 400 outstanding researchers and practitioners of federated learning are expected to attend the workshop.

 

(Please click here to find more

 

Thanks for supporting and using FATE. In the future, we will update the news monthly about the events of FATE. Let us jointly build the federated learning ecosystem!

 

Best regards !

Happy National Day of China

Tianjian Chen
 

Hi.All

 

A lot of members in FATE dev team and this community are Chinese. Happy National Day of China and enjoy your holidays.

 

Last month, the Linux Foundation officially announced FATE as a foundation project. We are moving forward to a more public and transparent governing structure.

https://www.linuxfoundation.org/press-release/2019/09/first-digital-only-bank-in-china-joins-linux-foundation/

 

Meanwhile, FATE 1.0.2 is released. This release contains many important updates, we suggest users upgrading to this version asap.

https://github.com/FederatedAI/FATE/releases/tag/v1.0.2

 

We are expecting a new website for FedAI and FATE in the late October and will start assembling the steering committee of FATE around the same time.

 

//bests

Toby

FATE updates now and release a significant bounty program for contributors!

FedAI Admin
 

Hi All,


Our contributor bounty program has been online : )
We are welcome you to check out our GitHub (https://github.com/FederatedAI/FATE) and submit pr or issue to join us as a contributor.



Our FATE version 1.0.2 is online now. The update is as follow:

 

# Release 1.0.2
## Major Features and Improvements
* Python and JDK environment are required only for running standalone version quick experiment
* Support cluster version docker deployment
* Add deployment guide in Chinese
* Standalone version job for quick experiment is supported when cluster version deployed.


If you have any questions about the contributor bounty program and FATE, please scan the QR code below to add our FATE official assistant for help:


Re: 答复: (Internet Mail)Re: [Fate-FedAI] Some questions and issues when depoy the fate

Tianjian Chen
 

 

Suggest using this FedAI Helper on Wechat

 

发件人: Fate-FedAI@groups.io <Fate-FedAI@groups.io> 代表 fave_toy@...
发送时间: 2019910 17:25
收件人: Fate-FedAI@groups.io
主题: (Internet Mail)Re: [Fate-FedAI] Some questions and issues when depoy the fate

 

hello!

I was trying to deploy a standalone FATE from the Github following the docker instructions from https://github.com/FederatedAI/FATE/tree/master/standalone-deploy

 

however, at the "docker exec -t -i ${CONTAINER_ID} bash" step, I was returned the error "Error response from daemon: Container xxx is not running". 

"xxx" is the long container id string 

 

thank you for your help!

Re: Some questions and issues when depoy the fate

fave_toy@...
 

hello!
I was trying to deploy a standalone FATE from the Github following the docker instructions from https://github.com/FederatedAI/FATE/tree/master/standalone-deploy
 
however, at the "docker exec -t -i ${CONTAINER_ID} bash" step, I was returned the error "Error response from daemon: Container xxx is not running". 
"xxx" is the long container id string 
 
thank you for your help!

Confusing ftl example

yao tc
 

Hi, I ran the ftl example with the plain model, and found the loss did not decrease after each iteration, and the final prediction did not make sense at all(With all predicted probabilities near 0.9). Could anyone give me some advice?

Btw, I used the default configuration at here.

 

Does FATE support multiple host training

liangnasty@...
 

Hi there,

First off, thanks for the great work!

I have a question regarding the examples, I wonder if FATE's runtime supports multiple hosts and how to extend the current examples to support that.

Best regards,
Marc

WeBank Held the 1st International Workshop on Federated Machine Learning in conjunction with IJCAI 2019

FedAI Admin
 

WeBank Held the 1st International Workshop on Federated Machine Learning in conjunction with IJCAI 2019

Once a concept, AI is now ushering in a key stage of application. What’s the solution to the data silos among businesses? Given the enhanced regulation on data at home and abroad, what’s the solution to data privacy and security concerns? What’s the status quo of Federated Machine Learning and how to establish an ecosystem for FML in the future?
 
WeBank, IBM and other organizations jointly held the 1st International Workshop on Federated Machine Learning for User Privacy and Data Confidentiality (FML’19) in conjunction with the 28th International Joint Conference on Artificial Intelligence (IJCAI-19) on 12th Aug. 2019, to further discussion on these issues.


President of IJCAI, Chair of FML Steering Committee, Chief AI Officer of WeBank Professor Qiang Yang delivered opening remarks at the workshop. Dr. Shahrokh Daijavad from IBM and Dr. Jakub Konečný from Google presented keynote addresses. In the panel discussion, top scholars from WeBank, Bar-Ilan University, IBM, Squirrel AI, Google, Huawei, Clustar, Sinovation Ventures and many other renowned enterprises and universities shared and discussed their findings and experience in FML as an emerging AI technology.

This workshop received 40 papers, of which 12 were presented during the workshop, 19 presented via poster. Awards include Best Theory Paper Award, Best Application Paper Award, Best Student Paper Award, Best Presentation Award. Selected high quality papers will be invited for publication in a special issue in the IEEE Intelligent Systems journal. All these attracted numerous scholars to engage in discussions and join efforts for building the FML ecosystem.
 
Experts from IBM and Google Share Groundbreaking Findings with a Focus on the Theory and Application of FML
 
Privacy and security are becoming a key concern in our digital age. On 25th May last year, the implementation of General Data Protection Regulation (GDPR) by the EU, the toughest Act on data privacy protection, stressed that user data collection must be open and transparent. A series of laws and regulations from China and overseas also pose new challenges to the traditional way of handling data and model for cooperation. Seeking ways for AI to adapt to this new reality became top priority, a demand that led to this workshop on FML.
 
A wealth of solutions and breakthroughs were shared by Dr. Shahrokh Daijavad from IBM and Dr. Jakub Konečný from Google in their speech on FML.
 
Besides how FML can help tackle challenges in the business world, Dr. Shahrokh Daijavad also shared the concept of Fusion AI, which means to train models on widely distributed data sets, but fuse them to produce one equivalent to what centralized training would yield. “Unlike traditional machine learning, in Fusion AI, model parameters are shared and data is not transferred, which makes Fusion AI model better than models that moving data centrally.” Given the widely distributed data, the development of Fusion AI and FML became ever important and imminent.
 
Dr. Shahrokh Daijavad delivering speech at the workshop
“FML enables machine learning engineers and data scientists to work productively with decentralized data with privacy by default.” said Dr. Jakub Konečný from Google. He also shared with us how FML works and its use cases at Google. In the case of Gboard, as on-device data is privacy sensitive or large or is more relevant than server-side proxy data, and labels can be inferred naturally via user interaction, the application of Federated RNN compared to prior n-gram model can increase the accuracy of next-word prediction by 24%, and the click rate of prediction strip by 10%.

Dr. Jakub Konečný delivering speech at the workshop
Major Figure Panelists Discuss the Way Ahead for FML
 
The moderator of the panel discussion, AI Principal Scientist of WeBank Dr. Lixin Fan joined panelists including Professor Benny Pinkas from Bar-Ilan University, Dr. Shahrokh Daijavad of IBM Academy of Technology, Chief Architect of Squirrel AI Dr. Richard Tong, Research Scientist of Google Dr. Jakub Konečný, Dr. Baofeng Zhang from CTO Office of CBG Software in Huawei, Executive VP of Clustar Dr. Junxue Zhang, VP of AI Institute in Sinovation Ventures Dr. Ji Feng and other experts in a host of in-depth exchanges with attendees, to shed light on the way ahead for FML.

Panel discussion of 1st International Workshop on Federated Machine Learning
Experts shared thoughts in the panel discussion on questions including but not limited to: How to meet the security and compliance requirements? Is there a way to extend the value of data while observing user privacy and data security? Given the classic trade-off between data regulation and development of AI, how to achieve the long-term goal of establishing a stable and win-win business ecosystem?
 
List of Award-Winners

Best Theory Paper Award, Best Application Paper Award, Best Student Paper Award and Best Presentation Award selected by all attendees were announced at the closing of the workshop.
 
Best Theory Paper Award: Preserving User Privacy for Machine Learning: Local Differential Privacy or Federated Machine Learning? By Huadi Zheng, Haibo Hu & Ziyang Han;

Best Application Paper Award: FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare. By Yiqiang Chen, Jindong Wang, Chaohui Yu, Wen Gao & Xin Qin;
 
Best Student Paper Award: Quantifying the Performance of Federated Transfer Learning. By Qinghe Jing, Weiyan Wang, Junxue Zhang, Han Tian & Kai Chen;
 
Best Presentation Award: Federated Generative Privacy. By Aleksei Triastcyn and Boi Faltings.
 
President of IJCAI, Chief AI Officer of WeBank Professor Qiang Yang, Chief Architect of Squirrel AI Dr. Richard Tong, VP of AI Institute in Sinovation Ventures Dr. Ji Feng presented the awards.





“The mission of this International Federated Machine Learning Workshop is to facilitate further understanding in the academia, business community as well as legal and regulatory institutions by promoting the establishment of FML ecosystem in the hope that more businesses will join and build a platform for students aspired to work in FML to find research teams that suit them.” said Professor Qiang Yang.
 
Held on 10th – 16th Aug. 2019 in Macao, China, IJCAI-19 is one of the leading International Academic Conference on AI, attracting over 3000 AI research personnel and experts. The 1st International Workshop on Federated Machine Learning (FML’19) was a highlight for experts joining this event. Visionaries in the academia and industrial sector expressed the willingness to be part of the effort for academic research, application of FML in the future, and the development and boom of AI ecosystem.

FATE v1.0 come soon,Begin your journey of visualization on FATE.

FedAI Admin
 

FATE v1.0 come soon.
This version includes two new products of FATE. Let's take a look at its new features:

Re: Some questions and issues when depoy the fate

FedAI Admin
 

Hi, Martin.

Thanks for your consultation.
Wechat Assistant has answered your question.

Best Regards
FedAI

Re: Some questions and issues when depoy the fate

Martin Cai
 

Thank you community.

I have another question. Does we have the some AUC with sklearn. I have tested, but looks like there are some gap between those two . Thanks.

 

B.R

Martin

 

 

From: bounce+33533+6+1877011+4057612@groups.io <bounce+33533+6+1877011+4057612@groups.io> On Behalf Of FedAI Admin
Sent: 2019
723 10:49
To: Fate-FedAI@groups.io
Subject: Re: [Fate-FedAI] Some questions and issues when depoy the fate

 

Hi Martin.

Sorry for this delayed response. Glad that you have solved the problem.
You may add our WeChat: FATEZS001, so we can respond in time if you have any questions.

Best Regards
FedAI

Re: Some questions and issues when depoy the fate

FedAI Admin
 

你好,

很开心能成为你的框架学习首选。
推荐添加小助手微信号:FATEZS001,方便做进一步沟通及交流。

祝好!

Federated learning facebook group

aviad.herman@...
 

Hello everyone,

Here is a Facebook group for research discussions regarding Federated Learning (but NOT about FATE implementation).

Feel free to join at:

https://www.facebook.com/groups/Federated.Learning/

Re: Some questions and issues when depoy the fate

Xihuanirelia@...
 

您好,我是第一次使用这种框架学习,请问可以留一个联系方式么,我想请教您一下该如何配置环境和系统。

Re: Some questions and issues when depoy the fate

FedAI Admin
 

Hi Martin.

Sorry for this delayed response. Glad that you have solved the problem.
You may add our WeChat: FATEZS001, so we can respond in time if you have any questions.

Best Regards
FedAI

Re: Some questions and issues when depoy the fate

Martin Cai
 

I think I have resolved the cluster deployment issue, it works now.  Do we have group like wechat or slack channel. Then we can quick discuss  or chat, Thanks.

 

 

Best Regards,

Martin

 

 

From: bounce+33533+4+1877011+4057612@groups.io <bounce+33533+4+1877011+4057612@groups.io> On Behalf Of Martin Cai
Sent: 2019
721 20:41
To: Fate-FedAI@groups.io
Subject: [Fate-FedAI] Some questions and issues when depoy the fate

 

Hello here,

 

It is Martin from Shanghai. We have some scenario need to do the demo and doing the evaluation to see if the opensource FATE is suitable for our AI cases. If yes, we would like to

I have success installed and deployed the standalone version and run the test cases, and getting out the AUC. Now , I am moving forward and deploying the cluster version. But I face some issues and have some questions. So I write this mail to the community group, hope can get some help or information. Thanks in advance.

Question list:

1.       I see in the wiki, we need to run host and guest job both, looks like they are directly connected. But do we need the arbiter as well in cluster mode.

2.       I am starting storage-service-cxx service, but it is failed due to Segmentation fault (core dumped) .  And I run the storage-service instead. So What is the different between storage-service/storage-service-cxx services.

3.       I see the storageType of redis is 3. and I dont see any data put into redis. What is the usage of the Redis.

 

 

Best Regards,

Martin

 

Some questions and issues when depoy the fate

Martin Cai
 

Hello here,

 

It is Martin from Shanghai. We have some scenario need to do the demo and doing the evaluation to see if the opensource FATE is suitable for our AI cases. If yes, we would like to

I have success installed and deployed the standalone version and run the test cases, and getting out the AUC. Now , I am moving forward and deploying the cluster version. But I face some issues and have some questions. So I write this mail to the community group, hope can get some help or information. Thanks in advance.

Question list:

1.       I see in the wiki, we need to run host and guest job both, looks like they are directly connected. But do we need the arbiter as well in cluster mode.

2.       I am starting storage-service-cxx service, but it is failed due to Segmentation fault (core dumped) .  And I run the storage-service instead. So What is the different between storage-service/storage-service-cxx services.

3.       I see the storageType of redis is 3. and I don’t see any data put into redis. What is the usage of the Redis.

 

Best Regards,

Martin