dataTranfer & clipboardData research

[Piotr Jasiun]

Check what is the current status of dataTranfer & clipboardData objects in drag and drop and copy and paste.

Specs:

• ​paste event
• ​drag and drop

[Piotr Jasiun]

W3C Documentation

Both drop and paste events can contain ​DataTransfer object which is most important for us.

• in copy, paste etc. events as evt.clipboardData,
• in dragstart, drop etc. events as evt.dataTransfer.

DataTransfer object supports both old and new API.

New API

Basically it is a list of items (dataTransfer.items). Every item can be a file or a string data in certain type. Data type might be a MIME type but it is not necessary. Theoretically it is possible to have mixed files and string data. I wrote "theoretically", because it is not defined how such data should be treated:

• if you drop 2 files you will get two items, both of them are "file", so they are two part of data,
• if you drop html you can get also two items, both of them are strings: one in text/html type, second one in text/plain, so this is the same data in two format (this is what we get in Blink).

Specification does not state if data should be treated as a one piece of data in multiple formats of multiple pieces of data. It depends on data. The only information we get is: "The drag data store item list, in the order that the items were added to the list; most recently added last.".

Blink is now the only engine which supports new API.

Old API

Old API is more complex. DataTransfer contains:

• types - an array of string e.g. ['plain/text','plain/html'] or Files?,
• files - an array of files, if files array contains any element then types contains 'Files' element,
• getData(format) - get string data, format is a string type from types array,
• setData(format, data) - set data by format.
• clearData(optional format) - remove data.

Old API works together with the new one so there are two ways to get data from DataTransfer object.

​IE support only "Text" and "URL" types.

[Piotr Jasiun]

1. Testing environment

I tested Drag & Drop and Copy & Paste I the following cases. You can download dnd-paste-sample.html​ and crayons.jpg​ to CKEditor's samples/ directory to test them.

HTML from outside of editable area

Source HTML:

eu odio.</p>
<img src="./crayons.jpg" width="30" height="30" />
<p>Aliquam <b>mi est</b>, interdum

HTML from inside of editable area

As above.

Text from textarea

Textatea in the same document as content editable area. Textarea contains:

foo bar

2 images from system file browser

Because file API is based on files array multiple file should works the same way as one file, but if it is not possible to drop/paste two files I also tested one file.

2 text files from system file browser

As above.

Part of image from MS Paint

Part of png/jpeg file opened with MS Paint. It is not possible to drag part of the image from Paints editing area so only copy and paste is possible. The same for Gimp.

Part of document from MS Word/Wordpad

Document containing formatted (bolded) text and image:

[b]foo[/b]
[image]
bar

Text from notepad

Like for Paint D&D is not possible.

Text from pdf (Adobe Reader XI)

Like for Paint D&D is not possible.

2. Results: Blink

2.1. Drag & Drop

Tested with Chrome (version 32) and Opera (version 19).

HTML from outside of editable area

Data in two types:

• text/plain: correct plain text

  eu odio.
  
  
  Aliquam mi est, interdum
  

• text/html: far to many html tags

  <p style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span class="Apple-converted-space"> </span>eu odio.</p><img src="http://ckeditor.dev/samples/local/crayons.jpg" width="30" height="30" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; display: inline !important; float: none;"></span><p style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">Aliquam<span class="Apple-converted-space"> </span><b>mi est</b>, interdum</p>
  

HTML from inside of editable area

As above.

Text from textarea

Data in text/plain.

2 images from system file browser

Chrome: Three items:

• text/uri-list with file paths
• 2 files.

If I let Chrome do native D&D I get:

<a href="file:///E:/Pictures/testing%20images/butterfly.jpg">file:///E:/Pictures/testing%20images/butterfly.jpg</a>

Opera: Two file items.

If I let Opera do native D&D browsers loaded page change to

file:///C:/Users/Public/Pictures/Sample%20Pictures/Tulips.jpg

2 text files from system file browser

As above.

Part of image from MS Paint

D&D is not possible.

Part of document from MS Word/Wordpad

• text/plain: correct plain text

  eu odio.
  
  
  Aliquam mi est, interdum
  

• text/html: far to many html tags

  <p class=MsoNormal>foo<span style='mso-fareast-language:PL;mso-no-proof:yes'><!--[if gte vml 1]><v:shapetype
  id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t"
  path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f">
  <v:stroke joinstyle="miter"/>
  <v:formulas>
   <v:f eqn="if lineDrawn pixelLineWidth 0"/>
   <v:f eqn="sum @0 1 0"/>
   <v:f eqn="sum 0 0 @1"/>
   <v:f eqn="prod @2 1 2"/>
   <v:f eqn="prod @3 21600 pixelWidth"/>
   <v:f eqn="prod @3 21600 pixelHeight"/>
   <v:f eqn="sum @0 0 1"/>
   <v:f eqn="prod @6 1 2"/>
   <v:f eqn="prod @7 21600 pixelWidth"/>
   <v:f eqn="sum @8 21600 0"/>
   <v:f eqn="prod @7 21600 pixelHeight"/>
   <v:f eqn="sum @10 21600 0"/>
  </v:formulas>
  <v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect"/>
  <o:lock v:ext="edit" aspectratio="t"/>
  </v:shapetype><v:shape id="Picture_x0020_1" o:spid="_x0000_i1025" type="#_x0000_t75"
  style='width:453pt;height:302.25pt;visibility:visible;mso-wrap-style:square'>
  <v:imagedata src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image001.jpg"
   o:title="butterfly"/>
   </v:shape><![endif]--><![if !vml]><img width=604 height=403
   src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image002.jpg"
  v:shapes="Picture_x0020_1"><![endif]></span><o:p></o:p></p>
  
  <p class=MsoNormal>bar<o:p></o:p></p>
  

If I try native D&D (both ondrop and ondragover events are not canceled) I get no image and error:

Not allowed to load local resource: file:///C:/Users/pjasiun/AppData/Local/Temp/msohtmlclip1/01/clip_image002.jpg

Text from notepad

D&D is not possible.

Text from pdf (Adobe Reader XI)

D&D is not possible.

2.2. Copy & paste

For Copy and Paste Clipboard object is available as evt.clipboardData, not evt.dataTranfer.

HTML from outside of editable area

The same as D&D.

HTML from inside of editable area

The same as D&D/above.

Text from textarea

The same as D&D.

2 images from system file browser

No files nor items available in clipboardData.

2 text files from system file browser

No files nor items available in clipboardData.

Part of image from MS Paint

One item which is file in "image/png" type (it does not matter if the source file was png, jpg or bmp).

For native drop I get nothing.

Part of document from MS Word/Wordpad

Data in three formats:

• text/plain,
• text/html - is similar to dropped,
• text/rtf.

If I pasted data into CKEditor I get:

<p>foo<img src="file:///C:\Users\pjasiun\AppData\Local\Temp\msohtmlclip1\01\clip_image002.jpg" style="height:403px; width:604px" /></p>

Text from notepad

Data in text/plain.

Text from pdf (Adobe Reader XI)

Data in two formats:

• text/plain,
• text/rtf.

Using native paste plain text data is inserted.

By the way: in Chrome exists additional method of DataTransferItem: webkitGetAsEntry next to the standard getAsFile and getAsString.

3. Results: Firefox

3.1. Drag & Drop

New D&D API is not available. Tested with version 26.

HTML from outside of editable area

Data in four types

• text/_moz_htmlcontext,
• text/_moz_htmlinfo,
• text/html: correct html

  <p>eu odio.</p>
  
  <img src="http://ckeditor.dev/samples/local/crayons.jpg" height="30" width="30">
  
  <p>Aliquam <b>mi est</b>, interdum</p>
  

• text/plain: correct plain text

  eu odio.
  
  
  Aliquam mi est, interdum
  

HTML from inside of editable area

As above.

Text from textarea

Data in text/plain.

2 images from system file browser

types contains 3 types:

• "application/x-moz-file",
• "text/x-moz-url",
• "Files".

files contains 2 file elements.

2 text files from system file browser

As above.

Part of image from MS Paint

D&D is not possible.

Part of document from MS Word/Wordpad

Data in text/plain:

Foo

bar

Text from notepad

D&D is not possible.

Text from pdf (Adobe Reader XI)

D&D is not possible.

3.2. Copy & paste

for Copy and Paste Clipboard object is available as evt.clipboardData, not evt.dataTranfer (as in Chrome).

HTML from outside of editable area

As D&D.

HTML from inside of editable area

As D&D.

Text from textarea

Data in text/plain.

2 images from system file browser

types contains 2 types:

• application/x-moz-file,
• Files.

files contains only 1 file elements.

2 text files from system file browser

As above.

Part of image from MS Paint

No data types nor files. But you can paste image using naive paste:

<img alt="" src="data:image/png;base64,iVBORw0K…"></img>

Part of document from MS Word/Wordpad

Object as for D&D, but you can paste html (with formatting attributes, without image) using native C&P.

Also using native paste, if you select only the image in Word you can also paste it (like for part of image from MS Paint).

Text from notepad

Data in text/plain.

Text from pdf (Adobe Reader XI)

Data in text/plain.

4. Results: IE

4.1. Drag & Drop

For IE I tested IE 11 and IE 8.

IE does not support new API. Also ​it support only two data types: "Text" and "URL".

IE8 does not contains types nor files arrays in dataTransfer, but methods getData(), setData() and clearData() exists so you can get Text data using getData( 'Text' ) like in IE 11.

HTML from outside of editable area

You can get text data using getData("Text"):

eu odio.

Aliquam mi est, interdum

You can not get dragged HTML using DataTransfer object.

IE11: 1 type in types: "Text".

HTML from inside of editable area

As above.

Text from textarea

Data in 1 type: "Text".

2 images from system file browser

IE11: Only "Files" data type.

2 file in files array.

On IE we need to cancel dragover event to handle files dropping. Otherwise page would be reloaded instead of calling drop event.

IE8: Drop event, but IE8 does not support file upload so getData("Text") return null and there are no files array.

2 text files from system file browser

As above.

Part of image from MS Paint

D&D is not possible.

Part of document from MS Word/Wordpad

You can get text data using getData("Text"):

foo

bar

IE11: 1 type in types: "Text".

If I drag single image from Word I get empty DataTransfer object (no types nor files).

Text from notepad

D&D is not possible.

Text from pdf (Adobe Reader XI)

D&D is not possible.

4.2. Copy & paste

On IE we do not have evt.clipboardData nor evt.dataTransfer but we can use window.clipboardData:

• evt.dataTransfer is null
• evt.clipboardData is undefined
• window.dataTransfer is undefined
• window.clipboardData is [object DataTransfer]

The following tests will use window.clipboardData.

Also ​Microsoft Clipboard sample does not work on IE11 nor on IE8 (it works on IE 10).

HTML from outside of editable area

You can get plain text data calling clipboardData.getData("Text").

Using native paste you can paste html.

IE11: types is null.

HTML from inside of editable area

As above.

Text from textarea

As above.

2 images from system file browser

No paste event, only onBeforePaste with no Text data nor files (on IE 11 files array exists, but contains no element). Also IE show security dialog then I use onBeforePaste.

With native paste event nothing happen.

2 text files from system file browser

As above.

Part of image from MS Paint

IE11: One file in png format (it does not matter what was origin extension).

Using native paste you can paste image as data:image/png;base64.

IE8: Paste event with no text nor file.

Nothing happens using native paste.

Part of document from MS Word/Wordpad

IE11: Plain text data (clipboardData.getData("Text")) and image file in files array.

Using native paste you can paste proper HTML with image as data:image/png;base64.

IE8: Text data using getData("Text"). Plain text with native paste.

Text from notepad

You can get text data calling clipboardData.getData("Text").

IE11: types is null.

Text from pdf (Adobe Reader XI)

As above.

Native paste will insert HTML.

[Piotr Jasiun]

setData & getData

I tested if it is possible to pass data from drag event to drop event and from copy to paste event using dataTransfer and clipboardData. I testes if it is possible to:

• pass text data (using setData( 'text', '...' ) or setData( 'plain/text', '...' )),
• use custom data type (using setData( 'custom', '...' ) or setData( 'custom/type', '...' )),
• overwrite date so browser will insert custom data on native drop or paste.
• it is not possible to handle drop event with dropbin, like paste event with pastebin. When drop event is called it is too late to move focus, move selection or create overall layer. Also even if we do so, we will lose drop position and need to use buggie moveToPoint method. Fortunately when drop method is called cursor is on drop position on IE.

Drag and Drop

	text	custom	overwrite
Chrome 32	yes	yes	setData( 'plain/html' ) on dragstart
Opera 19	yes	yes	setData( 'plain/html' ) on dragstart
Firefox 26	yes	yes	setData( 'plain/html' ) on dragstart
IE 11	yes	no	no
IE 10	yes	no	no
IE 9	yes	no	no
IE 8	yes	no	no

• for Firefox, Chrome and Opera text data equals text/plain data, so if you set both then second will overwrite first,
• on IE it is not possible to replaces dragged data; it happened in widget because of bug; if you put following code in contenteditable area and drag img you will be able to drop text data, but it works randomly and stop working is you delete any attribute.

  <div tabIndex=-1 contentEditable=false id="img">
  	<img src="./crayons.jpg" width=15 height=15>
  </div>
  

Copy and Paste

	text	custom	overwrite
Chrome 32	yes	yes	setData( 'plain/html' ) on copy
Opera 19	yes	yes	setData( 'plain/html' ) on copy
Firefox 26	yes	no!	setData( 'plain/html' ) on copy
IE 11	yes	no	setData( 'text' ) on copy
IE 10	yes	no	setData( 'text' ) on copy
IE 9	yes	no	setData( 'text' ) on copy
IE 8	yes	no	setData( 'text' ) on copy

• You should call preventDefault on copy event to be able paste you own data.
• Also IE seems to have security bug. If we call setData in copy event and not cancel that event then security dialog (asking use if he want allows script to read clipboard data) wont be shown and we can read clipboard whenever we want. It works like accepted dialog.

  droparea.oncopy = function ( evt ) {
  	window.clipboardData.setData("Text","some test text");
  };
  

[Piotr Jasiun]

Conclusion:

• All browsers support dataTransfer and clipboardData at some level. IE do it in its own way and nothing change in IE 11 (in IE 10 they added files support and this seems to be the only big change). In all browsers drop and paste works differently and different data you can read from dataTransfer and clipboardData.
• In all browsers we can:
  • read plain text data (in both drop and paste situations),
  • overwrite pasting data,
  • cancel drop/paste event,
  • pass some data from copy to paste and from drag to drop.

• In some (mostly IE) cases we can NOT:
  • read HTML from clipboard,
  • overwrite dropping data,
  • use custom data types,
  • drop data form image processor/pdf reader/notepad; such D&D is not supported by OS,
  • use ​dropzone attribute.

• In "modern" browsers (IE10+, Chrome, Firefox):
  • drop one or multiple files works pretty similar,
  • on some browser it is possible to paste image as HTML with source data.

• Drop/Paste event should be able to contains multitype data and files. In most situation we are able to read or simulate data in both html and plain formats. It should be similar to one of the W3C APIs (​new or old). It should be also backward compatible with our current ​paste event what might be a little complicated because it support different types but only one type at once.

• If I cancel drop event I also cancel drag event. So if you drag text form textarea and cancel drop event to handle it manually then text will be still in textarea after drop. The same for draging from Word etc.. On the other hand is not unified if data should stay in source app after Drag and Drop (ex. if you drag content form textarea in one Chromes tab to the textarea in the other tab text will not be removed from the source textarea), it depends on application.

[Piotr Jasiun]

I created some additional summary of research.

How it is possible to get Clipboard object?

	D&D	C&P
Blink	evt.dataTransfer (with .items)	evt.clipboardData (with .items)
Firefox	evt.dataTransfer	evt.clipboardData
IE 11	evt.dataTransfer	window.clipboardData
IE 8	evt.dataTransfer (no .types nor .files)	window.clipboardData (no .types nor .files)

Is it possible to read HTML?

	D&D	C&P
Blink	yes (with additional elements)	yes (with additional elements)
Firefox	yes	yes
IE 11	no	no
IE 8	no	no

Is it/How it is possible to get <img> with data:image/png;base64 in src?

	D&D	C&P
Blink	no	no
Firefox	native drop file	native paste form MS Paint or Word (image selected)
IE 11	no	native paste form MS Paint or Word
IE 8	no	no

Is it/How it is possible to get dropped/pasted files?

	D&D	C&P
Blink	yes, files from file system	yes, image from Paint, but files avabible only in items (files is empty)
Firefox	yes, files from file system	yes, files from file system (only 1 file)
IE 11	yes, files from file system	yes, image from Paint or from Word
IE 8	no	no

[Piotrek Koszuliński]

IE's moveToPoint method stability tester

[Piotrek Koszuliński]

See ​http://dev.ckeditor.com/ticket/11437#comment:14.

[Jakub Ś]

Uploading Shapes pasted from MS Word

IE11
Works.
When pasting whole word document, Chrome and FF see it as HTML and paths to images and shapes are local paths what causes security alerts. IE11 does nice conversion and all shapes can be seen as img tag with data:image string in src attribute thus images and shapes are uploaded without problems. What is more they are uploaded with their custom styles (Please try red box (with text inside) in attached shapes_test.docx).

FF
Pasting single shape works but for basic shapes only. Please have a look at attached shapes_test.docx. There are two shapes there: an arrow and red box. Arrow can be uploaded without problems while red box is seen as image with local path what prevents it from being uploaded.

Chrome
The same as in Firefox.
Chrome additionally sees "o:gfxdata" with a long string like in data:image. This is binary representation of graphics for the purposes of backward compatibility and is available only in Word 2007+. This data is zipped and base-64 encoded so perhaps something on server side could be used to unzip, decode it and extract correct shape.