Orange: Preprocess - Revision history

Onnowpurbo: /* Contoh */

2020-04-19T01:19:02Z

Contoh

Onnowpurbo: /* Contoh */

2020-04-19T01:16:18Z

Contoh

Onnowpurbo: /* Contoh */

2020-04-18T23:01:45Z

Contoh

Onnowpurbo at 07:30, 18 April 2020

2020-04-18T07:30:30Z

Onnowpurbo: /* Contoh */

2020-01-31T02:07:29Z

Contoh

Onnowpurbo at 01:56, 31 January 2020

2020-01-31T01:56:06Z

Onnowpurbo at 06:29, 27 January 2020

2020-01-27T06:29:09Z

Onnowpurbo at 01:52, 22 January 2020

2020-01-22T01:52:04Z

Onnowpurbo: Created page with "Sumber: https://docs.biolab.si//3/visual-programming/widgets/data/preprocess.html Preprocesses data with selected methods. Inputs Data: input dataset Outputs Pre..."

2020-01-11T00:42:52Z

Created page with "Sumber: https://docs.biolab.si//3/visual-programming/widgets/data/preprocess.html Preprocesses data with selected methods. Inputs Data: input dataset Outputs Pre..."

New page

Sumber: https://docs.biolab.si//3/visual-programming/widgets/data/preprocess.html

Preprocesses data with selected methods.

Inputs

Data: input dataset

Outputs

Preprocessor: preprocessing method

Preprocessed Data: data preprocessed with selected methods

Preprocessing is crucial for achieving better-quality analysis results. The Preprocess widget offers several preprocessing methods that can be combined in a single preprocessing pipeline. Some methods are available as separate widgets, which offer advanced techniques and greater parameter tuning.

../../_images/preprocess-stamped.png

List of preprocessors. Double click the preprocessors you wish to use and shuffle their order by dragging them up or down. You can also add preprocessors by dragging them from the left menu to the right.

Preprocessing pipeline.

When the box is ticked (Send Automatically), the widget will communicate changes automatically. Alternatively, click Send.

Preprocessors

../../_images/Preprocess1-stamped.png

Discretization of continuous values:

Entropy-MDL discretization by Fayyad and Irani that uses expected information to determine bins.

Equal frequency discretization splits by frequency (same number of instances in each bin.

Equal width discretization creates bins of equal width (span of each bin is the same).

Remove numeric features altogether.

Continuization of discrete values:

Most frequent as base treats the most frequent discrete value as 0 and others as 1. The discrete attributes with more than 2 values, the most frequent will be considered as a base and contrasted with remaining values in corresponding columns.

One feature per value creates columns for each value, place 1 where an instance has that value and 0 where it doesn’t. Essentially One Hot Encoding.

Remove non-binary features retains only categorical features that have values of either 0 or 1 and transforms them into continuous.

Remove categorical features removes categorical features altogether.

Treat as ordinal takes discrete values and treats them as numbers. If discrete values are categories, each category will be assigned a number as they appear in the data.

Divide by number of values is similar to treat as ordinal, but the final values will be divided by the total number of values and hence the range of the new continuous variable will be [0, 1].

Impute missing values:

Average/Most frequent replaces missing values (NaN) with the average (for continuous) or most frequent (for discrete) value.

Replace with random value replaces missing values with random ones within the range of each variable.

Remove rows with missing values.

Select relevant features:

Similar to Rank, this preprocessor outputs only the most informative features. Score can be determined by information gain, gain ratio, gini index, ReliefF, fast correlation based filter, ANOVA, Chi2, RReliefF, and Univariate Linear Regression.

Strategy refers to how many variables should be on the output. Fixed returns a fixed number of top scored variables, while Percentile return the selected top percent of the features.

../../_images/Preprocess2-stamped.png

Select random features outputs either a fixed number of features from the original data or a percentage. This is mainly used for advanced testing and educational purposes.

Normalize adjusts values to a common scale. Center values by mean or median or omit centering altogether. Similar for scaling, one can scale by SD (standard deviation), by span or not at all.

Randomize instances. Randomize classes shuffles class values and destroys connection between instances and class. Similarly, one can randomize features or meta data. If replicable shuffling is on, randomization results can be shared and repeated with a saved workflow. This is mainly used for advanced testing and educational purposes.

Remove sparse features retains features that have more than user-defined threshold percentage of non-zero values. The rest are discarded.

Principal component analysis outputs results of a PCA transformation. Similar to the PCA widget.

CUR matrix decomposition is a dimensionality reduction method, similar to SVD.

Examples

In the first example, we have used the heart_disease.tab dataset available in the dropdown menu of the File widget. then we used Preprocess to impute missing values and normalize features. We can observe the changes in the Data Table and compare it to the non-processed data.

../../_images/Preprocess-Example1.png

In the second example, we show how to use Preprocess for predictive modeling.

This time we are using the heart_disease.tab data from the File widget. You can access the data through the dropdown menu. This is a dataset with 303 patients that came to the doctor suffering from a chest pain. After the tests were done, some patients were found to have diameter narrowing and others did not (this is our class variable).

The heart disease data have some missing values and we wish to account for that. First, we will split the data set into train and test data with Data Sampler.

Then we will send the Data Sample into Preprocess. We will use Impute Missing Values, but you can try any combination of preprocessors on your data. We will send preprocessed data to Logistic Regression and the constructed model to Predictions.

Finally, Predictions also needs the data to predict on. We will use the output of Data Sampler for prediction, but this time not the Data Sample, but the Remaining Data, this is the data that wasn’t used for training the model.

Notice how we send the remaining data directly to Predictions without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to Test & Score.

../../_images/Predictions-Example2.png

==Referensi==

* https://docs.biolab.si//3/visual-programming/widgets/data/preprocess.html

==Pranala Menarik==

* [[Orange]]

@@ Line 72: / Line 72: @@
 Akhirnya, Prediksi juga membutuhkan data untuk diprediksi. Kita akan menggunakan output widget Data Sampler untuk prediksi, tetapi kali ini bukan Sample Data, tetapi Data Remaining (data test), ini adalah data yang tidak digunakan untuk men-train model.
-Perhatikan bagaimana kita mengirim data remaining (data test) secara langsung ke widget Prediksi tanpa menerapkan preprocessing apa pun. Ini karena Orange menangani preprocessing pada data baru secara internal untuk mencegah kesalahan dalam konstruksi model. Preprosesor yang sama persis yang digunakan pada data train akan digunakan untuk prediksi. Proses yang sama berlaku untuk widget Test & Score.
+Perhatikan bagaimana kita mengirim data remaining (data test) secara langsung ke widget Predictions tanpa menerapkan preprocessing apa pun. Ini karena Orange menangani preprocessing pada data baru secara internal untuk mencegah kesalahan dalam konstruksi model. Preprosesor yang sama persis yang digunakan pada data train akan digunakan untuk prediksi. Proses yang sama berlaku untuk widget Test & Score.
 [[File:Predictions-Example2.png|center|600px|thumb]]

@@ Line 68: / Line 68: @@
 Data penyakit jantung memiliki beberapa nilai yang hilang dan kita ingin menjelaskannya. Pertama, kita akan membagi dataset menjadi data train dan data test menggunakan widget Data Sampler.
-Kemudian kita akan mengirimkan Data Sample (data train) ke dalam widget Preprocess. Kita akan menggunakan Impute Missing Values, tentunya kita dapat mencoba berbagai kombinasi preprosesor pada data tersebut. Kita akan mengirimkan data yang telah diproses ke widget Logistic Regression dan model yang dibangun ke widget Prediction.
+Kemudian kita akan mengirimkan Data Sample (data train) ke dalam widget Preprocess. Kita akan menggunakan Impute Missing Values, tentunya kita dapat mencoba berbagai kombinasi preprosesor pada data tersebut. Kita akan mengirimkan data yang telah diproses ke widget Logistic Regression dan model yang dibangun ke widget Predictions.
 Akhirnya, Prediksi juga membutuhkan data untuk diprediksi. Kita akan menggunakan output widget Data Sampler untuk prediksi, tetapi kali ini bukan Sample Data, tetapi Data Remaining (data test), ini adalah data yang tidak digunakan untuk men-train model.

@@ Line 60: / Line 60: @@
 Pada contoh berikut ini, kita telah menggunakan dataset heart_disease.tab yang tersedia di dropdown menu dari widget File. Kemudian, kita menggunakan widget Preprocess untuk memasukkan nilai yang hilang dan menormalkan feature. Kita bisa mengamati perubahan yang terjadi di widget Data Table dan membandingkannya dengan data yang tidak diproses.
-[[File:Preprocess-Example1.png|center|200px|thumb]]
+[[File:Preprocess-Example1.png|center|600px|thumb]]
 Pada contoh berikut ini, kita akan menggunakan widget Preprocess untuk pemodelan prediktif.

@@ Line 1: / Line 1: @@
 Sumber: https://docs.biolab.si//3/visual-programming/widgets/data/preprocess.html
-Preprocess data dengan metode yang dipilih.
+Widget Preprocess akan mem-preprocess data dengan metode yang dipilih.
 ==Input==
@@ Line 12: / Line 12: @@
   Preprocessed Data: data preprocessed with selected methods
-Preprocessing sangat penting untuk mencapai hasil analisis berkualitas lebih baik. Preprocess widget menawarkan beberapa metode preprocessing yang dapat digabungkan dalam satu pipa preprocessing tunggal. Beberapa metode tersedia sebagai widget terpisah, yang menawarkan teknik canggih dan penyetelan parameter yang lebih banyak.
+Preprocessing sangat penting untuk mencapai hasil analisis berkualitas lebih baik. Widget Preprocess menawarkan beberapa metode preprocessing yang dapat digabungkan dalam satu pipa preprocessing tunggal. Beberapa metode tersedia sebagai widget terpisah, yang menawarkan teknik canggih dan setup parameter yang lebih banyak.
-[[File:Preprocess-stamped.png|center|200px|thumb]]
+[[File:Preprocess-stamped.png|center|400px|thumb]]
 * List of preprocessors. Double click the preprocessors you wish to use and shuffle their order by dragging them up or down. You can also add preprocessors by dragging them from the left menu to the right.
@@ Line 58: / Line 58: @@
 ==Contoh==
-Pada contoh pertama, kita telah menggunakan dataset heart_disease.tab yang tersedia di dropdown menu dari File widget. lalu kita menggunakan Preprocess untuk memasukkan nilai yang hilang dan menormalkan fitur. Kita bisa mengamati perubahan dalam Tabel Data dan membandingkannya dengan data yang tidak diproses.
+Pada contoh berikut ini, kita telah menggunakan dataset heart_disease.tab yang tersedia di dropdown menu dari widget File. Kemudian, kita menggunakan widget Preprocess untuk memasukkan nilai yang hilang dan menormalkan feature. Kita bisa mengamati perubahan yang terjadi di widget Data Table dan membandingkannya dengan data yang tidak diproses.
 [[File:Preprocess-Example1.png|center|200px|thumb]]
-Pada contoh kedua, kami menunjukkan cara menggunakan Preprocess untuk pemodelan prediktif.
+Pada contoh berikut ini, kita akan menggunakan widget Preprocess untuk pemodelan prediktif.
-Kali ini kita menggunakan data heart_disease.tab dari File widget. Kita dapat mengakses data melalui menu dropdown. Ini adalah dataset dengan 303 pasien yang datang ke dokter yang menderita nyeri dada. Setelah tes dilakukan, beberapa pasien ditemukan memiliki penyempitan diameter dan yang lainnya tidak (ini adalah variabel kelas kita).
+Kali ini kita menggunakan dataset heart_disease.tab dari widget File. Kita dapat mengakses data melalui menu dropdown. Ini adalah dataset dengan 303 pasien yang datang ke dokter yang menderita nyeri dada. Setelah tes dilakukan, beberapa pasien ditemukan memiliki penyempitan diameter dan yang lainnya tidak (ini adalah variabel class).
-Data penyakit jantung memiliki beberapa nilai yang hilang dan kita ingin menjelaskannya. Pertama, kita akan membagi set data menjadi data train dan data test dengan Data Sampler.
+Data penyakit jantung memiliki beberapa nilai yang hilang dan kita ingin menjelaskannya. Pertama, kita akan membagi dataset menjadi data train dan data test menggunakan widget Data Sampler.
-Kemudian kita akan mengirimkan Sampel Data ke dalam Preprocess. Kita akan menggunakan Impute Missing Values, tetapi kita dapat mencoba berbagai kombinasi preprosesor pada data kita. Kita akan mengirimkan data yang telah diproses ke Logistic Regression dan model yang dibangun ke Prediction.
+Kemudian kita akan mengirimkan Data Sample (data train) ke dalam widget Preprocess. Kita akan menggunakan Impute Missing Values, tentunya kita dapat mencoba berbagai kombinasi preprosesor pada data tersebut. Kita akan mengirimkan data yang telah diproses ke widget Logistic Regression dan model yang dibangun ke widget Prediction.
-Akhirnya, Prediksi juga membutuhkan data untuk diprediksi. Kita akan menggunakan output Data Sampler untuk prediksi, tetapi kali ini bukan Sampel Data, tetapi Data sisa, ini adalah data yang tidak digunakan untuk men-train model.
+Akhirnya, Prediksi juga membutuhkan data untuk diprediksi. Kita akan menggunakan output widget Data Sampler untuk prediksi, tetapi kali ini bukan Sample Data, tetapi Data Remaining (data test), ini adalah data yang tidak digunakan untuk men-train model.
-Perhatikan bagaimana kita mengirim data yang tersisa secara langsung ke Prediksi tanpa menerapkan preprocessing apa pun. Ini karena Orange menangani preprocessing pada data baru secara internal untuk mencegah kesalahan dalam konstruksi model. Preprosesor yang sama persis yang digunakan pada data train akan digunakan untuk prediksi. Proses yang sama berlaku untuk Test & Score.
+Perhatikan bagaimana kita mengirim data remaining (data test) secara langsung ke widget Prediksi tanpa menerapkan preprocessing apa pun. Ini karena Orange menangani preprocessing pada data baru secara internal untuk mencegah kesalahan dalam konstruksi model. Preprosesor yang sama persis yang digunakan pada data train akan digunakan untuk prediksi. Proses yang sama berlaku untuk widget Test & Score.
-[[File:Predictions-Example2.png|center|200px|thumb]]
+[[File:Predictions-Example2.png|center|600px|thumb]]
 ==Referensi==

@@ Line 68: / Line 68: @@
 Data penyakit jantung memiliki beberapa nilai yang hilang dan kita ingin menjelaskannya. Pertama, kita akan membagi set data menjadi data train dan data test dengan Data Sampler.
-Then we will send the Data Sample into Preprocess. We will use Impute Missing Values, but you can try any combination of preprocessors on your data. We will send preprocessed data to Logistic Regression and the constructed model to Predictions.
+Kemudian kita akan mengirimkan Sampel Data ke dalam Preprocess. Kita akan menggunakan Impute Missing Values, tetapi kita dapat mencoba berbagai kombinasi preprosesor pada data kita. Kita akan mengirimkan data yang telah diproses ke Logistic Regression dan model yang dibangun ke Prediction.
-Finally, Predictions also needs the data to predict on. We will use the output of Data Sampler for prediction, but this time not the Data Sample, but the Remaining Data, this is the data that wasn’t used for training the model.
+Akhirnya, Prediksi juga membutuhkan data untuk diprediksi. Kita akan menggunakan output Data Sampler untuk prediksi, tetapi kali ini bukan Sampel Data, tetapi Data sisa, ini adalah data yang tidak digunakan untuk men-train model.
-Notice how we send the remaining data directly to Predictions without applying any preprocessing. This is because Orange handles preprocessing on new data internally to prevent any errors in the model construction. The exact same preprocessor that was used on the training data will be used for predictions. The same process applies to Test & Score.
+Perhatikan bagaimana kita mengirim data yang tersisa secara langsung ke Prediksi tanpa menerapkan preprocessing apa pun. Ini karena Orange menangani preprocessing pada data baru secara internal untuk mencegah kesalahan dalam konstruksi model. Preprosesor yang sama persis yang digunakan pada data train akan digunakan untuk prediksi. Proses yang sama berlaku untuk Test & Score.
 [[File:Predictions-Example2.png|center|200px|thumb]]
 ==Referensi==

@@ Line 6: / Line 6: @@
 ==Input==
-    Data: input dataset
+ Data: input dataset
 ==Output==
-    Preprocessor: preprocessing method
+ Preprocessor: preprocessing method
-    Preprocessed Data: data preprocessed with selected methods
+ Preprocessed Data: data preprocessed with selected methods
 Preprocessing is crucial for achieving better-quality analysis results. The Preprocess widget offers several preprocessing methods that can be combined in a single preprocessing pipeline. Some methods are available as separate widgets, which offer advanced techniques and greater parameter tuning.
@@ Line 17: / Line 17: @@
 [[File:Preprocess-stamped.png|center|200px|thumb]]
-    List of preprocessors. Double click the preprocessors you wish to use and shuffle their order by dragging them up or down. You can also add preprocessors by dragging them from the left menu to the right.
+* List of preprocessors. Double click the preprocessors you wish to use and shuffle their order by dragging them up or down. You can also add preprocessors by dragging them from the left menu to the right.
-    Preprocessing pipeline.
+==Preprocessors==
 [[File:Preprocess1-stamped.png|center|200px|thumb]]
-    Discretization of continuous values:
+Discretization of continuous values:
+* Entropy-MDL discretization by Fayyad and Irani that uses expected information to determine bins.
-        Entropy-MDL discretization by Fayyad and Irani that uses expected information to determine bins.
+* Equal frequency discretization splits by frequency (same number of instances in each bin.
-        Equal frequency discretization splits by frequency (same number of instances in each bin.
+* Equal width discretization creates bins of equal width (span of each bin is the same).
-        Equal width discretization creates bins of equal width (span of each bin is the same).
+* Remove numeric features altogether.
-        Remove non-binary features retains only categorical features that have values of either 0 or 1 and transforms them into continuous.
+Continuization of discrete values:
-        Remove categorical features removes categorical features altogether.
+Impute missing values:
-        Treat as ordinal takes discrete values and treats them as numbers. If discrete values are categories, each category will be assigned a number as they appear in the data.
+Select relevant features:
+* Similar to Rank, this preprocessor outputs only the most informative features. Score can be determined by information gain, gain ratio, gini index, ReliefF, fast correlation based filter, ANOVA, Chi2, RReliefF, and Univariate Linear Regression.
-        Divide by number of values is similar to treat as ordinal, but the final values will be divided by the total number of values and hence the range of the new continuous variable will be [0, 1].
+* Strategy refers to how many variables should be on the output. Fixed returns a fixed number of top scored variables, while Percentile return the selected top percent of the features.
 [[File:Preprocess2-stamped.png|center|200px|thumb]]
-    Select random features outputs either a fixed number of features from the original data or a percentage. This is mainly used for advanced testing and educational purposes.
+* Select random features outputs either a fixed number of features from the original data or a percentage. This is mainly used for advanced testing and educational purposes.
+* Normalize adjusts values to a common scale. Center values by mean or median or omit centering altogether. Similar for scaling, one can scale by SD (standard deviation), by span or not at all.
-    Normalize adjusts values to a common scale. Center values by mean or median or omit centering altogether. Similar for scaling, one can scale by SD (standard deviation), by span or not at all.
+* Randomize instances. Randomize classes shuffles class values and destroys connection between instances and class. Similarly, one can randomize features or meta data. If replicable shuffling is on, randomization results can be shared and repeated with a saved workflow. This is mainly used for advanced testing and educational purposes.
+* Remove sparse features retains features that have more than user-defined threshold percentage of non-zero values. The rest are discarded.
-    Randomize instances. Randomize classes shuffles class values and destroys connection between instances and class. Similarly, one can randomize features or meta data. If replicable shuffling is on, randomization results can be shared and repeated with a saved workflow. This is mainly used for advanced testing and educational purposes.
+* Principal component analysis outputs results of a PCA transformation. Similar to the PCA widget.
+* CUR matrix decomposition is a dimensionality reduction method, similar to SVD.
-Examples
+==Contoh==
 In the first example, we have used the heart_disease.tab dataset available in the dropdown menu of the File widget. then we used Preprocess to impute missing values and normalize features. We can observe the changes in the Data Table and compare it to the non-processed data.